From 48e6161eddd599f1d9cf21b9b3371facafd798e4 Mon Sep 17 00:00:00 2001 From: Mason Fox Date: Thu, 5 Feb 2026 15:32:22 -0500 Subject: [PATCH 01/89] spec: add non-Calibre books specification with clarifications (spec-003) - Create spec-003 for supporting manual and external provider books - Add comprehensive user stories for manual book addition, sync isolation, filtering, and external metadata integration - Include clarifications session covering uniqueness enforcement, API timeouts, form validation, rate limiting, and page count validation - Define 12 functional requirements with validation rules - Specify measurable success criteria and edge case handling - Document assumptions, dependencies, and out-of-scope items --- .../checklists/requirements.md | 86 +++++++++ specs/003-non-calibre-books/spec.md | 178 ++++++++++++++++++ 2 files changed, 264 insertions(+) create mode 100644 specs/003-non-calibre-books/checklists/requirements.md create mode 100644 specs/003-non-calibre-books/spec.md diff --git a/specs/003-non-calibre-books/checklists/requirements.md b/specs/003-non-calibre-books/checklists/requirements.md new file mode 100644 index 00000000..a8b3b9c0 --- /dev/null +++ b/specs/003-non-calibre-books/checklists/requirements.md @@ -0,0 +1,86 @@ +# Specification Quality Checklist: Support Non-Calibre Books + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2026-02-05 +**Feature**: [spec.md](../spec.md) + +## Content Quality + +- [x] No implementation details (languages, frameworks, APIs) +- [x] Focused on user value and business needs +- [x] Written for non-technical stakeholders +- [x] All mandatory sections completed + +## Requirement Completeness + +- [x] No [NEEDS CLARIFICATION] markers remain +- [x] Requirements are testable and unambiguous +- [x] Success criteria are measurable +- [x] Success criteria are technology-agnostic (no implementation details) +- [x] All acceptance scenarios are defined +- [x] Edge cases are identified +- [x] Scope is clearly bounded +- [x] Dependencies and assumptions identified + +## Feature Readiness + +- [x] All functional requirements have clear acceptance criteria +- [x] User scenarios cover primary flows +- [x] Feature meets measurable outcomes defined in Success Criteria +- [x] No implementation details leak into specification + +## Validation Results + +**Status**: ✅ PASSED - All checklist items complete + +### Content Quality Assessment + +1. **No implementation details**: Specification focuses on WHAT and WHY without mentioning specific frameworks, languages, or code structure. References to database schema changes in GitHub issue were abstracted to entity requirements. + +2. **User value focused**: All user stories clearly articulate the value proposition (e.g., "single source of truth for reading", "data integrity", "improved organization"). + +3. **Non-technical language**: Specification is written for business stakeholders with clear acceptance scenarios using Given/When/Then format. + +4. **Mandatory sections complete**: All required sections (User Scenarios, Requirements, Success Criteria) are fully populated. + +### Requirement Completeness Assessment + +1. **No clarification markers**: All requirements are concrete and specific. No [NEEDS CLARIFICATION] markers present. + +2. **Testable requirements**: Each functional requirement is verifiable (e.g., FR-001 can be tested by checking nullable Calibre ID, FR-006 can be tested by accessing manual book creation interface). + +3. **Measurable success criteria**: All success criteria include specific metrics: + - SC-001: "within 2 minutes" + - SC-002: "100% isolation" + - SC-005: "90%+ users understand" + - SC-006: "under 3 seconds" + - SC-007: "70% reduction" + +4. **Technology-agnostic success criteria**: No mention of implementation technologies in success criteria. All outcomes described from user/business perspective (e.g., "users can complete task" rather than "API responds in X ms"). + +5. **Acceptance scenarios defined**: Four prioritized user stories with comprehensive Given/When/Then scenarios covering manual book addition, sync isolation, filtering, and external metadata integration. + +6. **Edge cases identified**: Five key edge cases addressed including duplicate books, provider unavailability, orphaned book distinction, and empty Calibre library scenarios. + +7. **Scope bounded**: Clear "Out of Scope" section defining what will NOT be included (automatic duplicate detection, bulk import, multi-user access, etc.). + +8. **Dependencies and assumptions documented**: + - Dependencies: External metadata provider API for P3, existing Tome architecture + - Assumptions: User understanding, duplicate handling, minimum required fields, performance expectations + +### Feature Readiness Assessment + +1. **Functional requirements with acceptance criteria**: All 12 functional requirements have clear, testable acceptance criteria through the user story scenarios. + +2. **User scenarios cover primary flows**: Four prioritized user stories (P1-P3) cover the complete feature journey from basic manual entry to advanced filtering and external integration. + +3. **Measurable outcomes defined**: Seven success criteria provide concrete metrics for feature validation. + +4. **No implementation leakage**: Specification maintains abstraction throughout, avoiding technical implementation details. + +## Notes + +- Specification is ready for `/speckit.plan` or `/speckit.clarify` +- All validation criteria met on first pass +- Feature has clear phased approach (P1 → P2 → P3) enabling incremental delivery +- GitHub issue technical details successfully abstracted to business requirements diff --git a/specs/003-non-calibre-books/spec.md b/specs/003-non-calibre-books/spec.md new file mode 100644 index 00000000..62ecd953 --- /dev/null +++ b/specs/003-non-calibre-books/spec.md @@ -0,0 +1,178 @@ +# Feature Specification: Support Non-Calibre Books + +**Feature Branch**: `003-non-calibre-books` +**Created**: 2026-02-05 +**Status**: Draft +**Input**: User description: "I'd like begin preparing for this feature in a new branch: https://github.com/masonfox/tome/issues/185. This will be spec-003" + +## Clarifications + +### Session 2026-02-05 + +- Q: When a user manually adds a book, how should the system enforce uniqueness to prevent accidental duplicates? → A: Warn but allow - show warning if title+author match exists, let user proceed +- Q: When fetching metadata from external providers (like Hardcover), what should happen if the API request times out or takes too long? → A: 5-second timeout, automatically fallback to manual entry form +- Q: When a user is filling out the manual book form, should fields be validated in real-time (as they type) or only when they submit the form? → A: Real-time + submit - validate as user types and again on submit +- Q: If the external metadata provider (like Hardcover) has rate limits, how should the system handle exceeding those limits? → A: Graceful fallback with message - fallback to manual entry, show informative message +- Q: For the page count field in manual book creation, what validation rules should apply? → A: Positive integer only with reasonable maximum (1-10000 pages) + +## User Scenarios & Testing *(mandatory)* + +### User Story 1 - Manual Book Addition (Priority: P1) + +As a reader with physical books, I want to manually add books to my Tome library so that I can track my reading progress for both digital and physical books in one place. + +**Why this priority**: This is the core value proposition - enabling users who mix physical and digital books to have a single source of truth for their reading. Without this, users cannot track physical books at all. + +**Independent Test**: Can be fully tested by adding a manual book entry through the UI, logging progress for it, and viewing it in the library alongside Calibre-sourced books. Delivers immediate value by allowing tracking of non-Calibre books. + +**Acceptance Scenarios**: + +1. **Given** I'm viewing my library, **When** I click "Add Manual Book" and enter book details (title, author, page count), **Then** the book appears in my library with a visual indicator showing it's a manual entry +5. **Given** I'm filling out the manual book form, **When** I type in a required field (title, author, or page count), **Then** the system validates the field in real-time and displays any validation errors immediately +6. **Given** I've filled out the manual book form with invalid data, **When** I attempt to submit, **Then** the system validates all fields again and prevents submission until all required fields are valid +7. **Given** I'm entering page count in the manual book form, **When** I enter a value less than 1 or greater than 10000, **Then** the system displays a validation error indicating page count must be between 1 and 10000 +4. **Given** I'm adding a manual book with title and author matching an existing book, **When** I submit the form, **Then** system displays a warning about potential duplication but allows me to proceed with creation +2. **Given** I have manually added a book, **When** I log reading progress for that book, **Then** progress is saved and displayed just like Calibre books +3. **Given** I have both Calibre and manual books in my library, **When** I view my library, **Then** both types are displayed together with clear visual differentiation (source badges) + +--- + +### User Story 2 - Library Sync Isolation (Priority: P1) + +As a user with mixed book sources, I want Calibre syncs to only affect Calibre-sourced books so that my manually added books are never accidentally removed or modified during sync operations. + +**Why this priority**: Critical for data integrity. Without proper sync isolation, manual books could be orphaned or removed during Calibre sync, destroying user data and trust in the system. + +**Independent Test**: Can be tested by adding manual books, running a Calibre sync that removes books, and verifying manual books remain untouched. Delivers value by ensuring data safety. + +**Acceptance Scenarios**: + +1. **Given** I have manual books in my library, **When** a Calibre sync removes books from Calibre, **Then** only Calibre-sourced books are marked as orphaned, manual books remain active +2. **Given** I have manual books in my library, **When** a Calibre sync adds or updates books, **Then** manual books are unchanged and unaffected +3. **Given** I have a book with the same title in both Calibre and manual sources, **When** sync occurs, **Then** both books remain as separate entities with different source identifiers + +--- + +### User Story 3 - Source-Based Filtering and Display (Priority: P2) + +As a user managing multiple book sources, I want to filter my library by source (Calibre vs Manual) so that I can quickly view subsets of my collection based on how I obtained the books. + +**Why this priority**: Enhances usability for power users but isn't essential for core functionality. Users can still use the feature effectively without filtering. + +**Independent Test**: Can be tested by adding books from different sources, applying source filters, and verifying correct subset display. Delivers value through improved organization. + +**Acceptance Scenarios**: + +1. **Given** I have books from both Calibre and manual sources, **When** I apply a "Calibre only" filter, **Then** only Calibre-sourced books are displayed +2. **Given** I have books from multiple sources, **When** I apply a "Manual only" filter, **Then** only manually added books are displayed +3. **Given** I have filtered by source, **When** I clear the filter, **Then** all books from all sources are displayed again + +--- + +### User Story 4 - External Metadata Provider Integration (Priority: P3) + +As a user adding manual books, I want the system to optionally fetch metadata from external providers like Hardcover so that I don't have to manually enter all book details. + +**Why this priority**: Quality-of-life enhancement that reduces friction but isn't required for core functionality. Users can still manually enter all details if needed. + +**Independent Test**: Can be tested by initiating a manual book add, searching an external provider, selecting a book, and verifying metadata population. Delivers value through improved user experience. + +**Acceptance Scenarios**: + +1. **Given** I'm adding a manual book, **When** I search for the book title in the external provider search, **Then** matching books are displayed with cover images and basic metadata +4. **Given** I'm searching for a book in an external provider, **When** the API request exceeds 5 seconds, **Then** the system automatically falls back to the manual entry form and notifies me that external search is unavailable +5. **Given** I'm searching for a book in an external provider, **When** the provider's rate limit is exceeded, **Then** the system automatically falls back to the manual entry form and displays an informative message that the search service is temporarily unavailable +2. **Given** I've searched for a book in an external provider, **When** I select a book from results, **Then** title, author, page count, and cover image are auto-populated in the form +3. **Given** I've auto-populated book details from an external provider, **When** I edit any field before saving, **Then** my manual edits override the fetched metadata + +--- + +### Edge Cases + +- What happens when a user manually adds a book that already exists in their Calibre library (same title and author)? System checks for title+author matches across all sources, displays a warning showing the existing book(s) and their source(s), but allows the user to proceed with creation after acknowledging the warning. +- How does the system handle manual books when a user later adds the same book to Calibre? Both books remain separate; the user can optionally merge or delete one. +- What happens if an external metadata provider is unavailable or returns no results? System implements a 5-second timeout for API requests. If the provider is unavailable, times out, returns no results, or returns a rate limit error, the system automatically transitions to the manual entry form and displays an informative notification explaining why external search is unavailable. +- How are orphaned Calibre books distinguished from manual books in the UI? Orphaned books have a distinct "orphaned" indicator separate from the source badge. +- What happens when syncing with an empty Calibre library? Manual books remain untouched; only Calibre-sourced books would be orphaned. +- What happens if a user tries to submit the manual book form with missing required fields? Real-time validation displays error messages as the user types in each field. On submission attempt, the system validates all fields again and prevents form submission, highlighting all invalid or empty required fields until corrected. +- What happens if a user enters an invalid page count (e.g., 0, negative, or >10000)? The system displays a real-time validation error indicating that page count must be a positive integer between 1 and 10000, and prevents form submission until corrected. + +## Requirements *(mandatory)* + +### Functional Requirements + +- **FR-001**: System MUST allow Calibre ID to be null for books, enabling books without Calibre sources +- **FR-002**: System MUST track the source of each book (Calibre, Manual, or External Provider like Hardcover) +- **FR-003**: System MUST store external provider IDs for books sourced from external metadata providers +- **FR-004**: System MUST restrict sync operations to only affect books where source equals 'calibre' +- **FR-005**: System MUST NOT orphan or remove manual books during Calibre sync operations +- **FR-006**: Users MUST be able to create books manually through a dedicated interface +- **FR-007**: Manual book creation MUST collect at minimum: title, author, and page count +- **FR-007a**: System MUST check for existing books with matching title and author during manual book creation and display a warning to the user, but MUST allow creation to proceed if user confirms +- **FR-007b**: System MUST validate required fields (title, author, page count) in real-time as the user types and perform final validation on form submission, preventing submission if any required field is invalid or empty +- **FR-007c**: System MUST validate page count as a positive integer with a minimum value of 1 and maximum value of 10000, displaying validation errors for values outside this range +- **FR-008**: System MUST display visual indicators (badges/icons) distinguishing book sources in the library +- **FR-009**: System MUST allow all existing Tome features (progress tracking, sessions, goals, streaks) to work identically for manual and Calibre books +- **FR-010**: System MUST provide UI access to add manual books from the main library view +- **FR-011**: System MUST support optional metadata search from external providers during manual book creation +- **FR-011a**: System MUST implement a 5-second timeout for external metadata provider API requests and automatically fallback to manual entry form when timeout is reached or provider is unavailable +- **FR-011b**: System MUST handle external provider rate limit errors by automatically falling back to manual entry form and displaying an informative message to the user that the search service is temporarily unavailable +- **FR-012**: System MUST allow filtering library view by book source (optional for P2) + +### Key Entities + +- **Book (Extended)**: Represents any tracked book regardless of source + - Calibre ID (now optional/nullable): Links to Calibre database when source is 'calibre' + - Source: Identifies origin ('calibre', 'manual', 'hardcover') + - External ID: Stores provider-specific ID for books from external services + - Page Count: Positive integer with valid range 1-10000 + - All existing attributes (title, author, status, dates, etc.) remain unchanged + +- **Sync Operation (Behavioral Change)**: Calibre library synchronization + - Only processes books where source equals 'calibre' + - Only orphans books where source equals 'calibre' + - Ignores manual and external provider books completely + +## Success Criteria *(mandatory)* + +### Measurable Outcomes + +- **SC-001**: Users can successfully add a manual book and log progress within 2 minutes of first attempt +- **SC-002**: Calibre sync operations complete without affecting manual books (100% isolation verified through testing) +- **SC-003**: Users with mixed book sources report successful tracking of both physical and digital books in post-feature feedback +- **SC-004**: Manual books support all existing Tome features (sessions, progress, goals, streaks) with identical functionality to Calibre books +- **SC-005**: Library view clearly distinguishes book sources with visual indicators that 90%+ of users understand without explanation (based on user testing) +- **SC-006**: Book filtering by source (when implemented) allows users to view source-specific subsets in under 3 seconds +- **SC-007**: External metadata provider integration (when implemented) reduces manual data entry time by 70% for users who use the feature +- **SC-008**: External metadata provider requests that exceed 5 seconds automatically fallback to manual entry without user intervention 100% of the time +- **SC-009**: External metadata provider rate limit errors result in graceful fallback to manual entry with informative messaging 100% of the time + +## Assumptions + +- Users understand the difference between Calibre-sourced and manually added books +- Most users mixing physical and digital books will want to see both in a unified library view by default +- Duplicate books (same title/author from different sources) are acceptable; users can manage duplicates themselves +- Manual books do not require ISBN or other formal identifiers; title/author/pages are sufficient minimums +- External metadata providers will use standard REST APIs accessible from the Tome backend +- The Hardcover API (if used) provides adequate metadata (title, author, pages, cover) without authentication requirements +- Sync performance will not degrade significantly with mixed-source libraries of typical size (under 10,000 books) +- Visual source indicators (badges/icons) are sufficient differentiation; color-coding is not required +- Users do not expect manual books to retroactively sync with Calibre if later added to Calibre library + +## Dependencies + +- No external service dependencies for core manual book functionality (P1) +- External metadata provider API (e.g., Hardcover) required for P3 auto-population feature +- Existing Tome architecture (database, repositories, sync service) must remain intact +- Calibre database structure remains unchanged (read-only except ratings) + +## Out of Scope + +- Automatic detection of duplicate books across sources (user must manually identify) +- Migration or merging of manual books into Calibre library +- Bulk import of manual books from CSV or other formats +- Advanced metadata fields beyond core requirements (ISBN, publisher, publication date optional for future) +- Multi-user access or sharing of manual books +- Export of manual books to other systems (beyond existing Tome export capabilities) +- Editing or updating books in Calibre database from manual entries +- Automatic matching between manual books and later-added Calibre books From fe9d1fef510cdf42147ab01f1bbade820d29c554 Mon Sep 17 00:00:00 2001 From: Mason Fox Date: Thu, 5 Feb 2026 18:07:47 -0500 Subject: [PATCH 02/89] enhance spec --- specs/003-non-calibre-books/spec.md | 383 ++++++++++++++++++++++------ 1 file changed, 312 insertions(+), 71 deletions(-) diff --git a/specs/003-non-calibre-books/spec.md b/specs/003-non-calibre-books/spec.md index 62ecd953..5efc3916 100644 --- a/specs/003-non-calibre-books/spec.md +++ b/specs/003-non-calibre-books/spec.md @@ -7,7 +7,7 @@ ## Clarifications -### Session 2026-02-05 +### Session 2026-02-05 (Initial) - Q: When a user manually adds a book, how should the system enforce uniqueness to prevent accidental duplicates? → A: Warn but allow - show warning if title+author match exists, let user proceed - Q: When fetching metadata from external providers (like Hardcover), what should happen if the API request times out or takes too long? → A: 5-second timeout, automatically fallback to manual entry form @@ -15,6 +15,33 @@ - Q: If the external metadata provider (like Hardcover) has rate limits, how should the system handle exceeding those limits? → A: Graceful fallback with message - fallback to manual entry, show informative message - Q: For the page count field in manual book creation, what validation rules should apply? → A: Positive integer only with reasonable maximum (1-10000 pages) +### Session 2026-02-05 (Architecture Decisions) + +- Q: How should providers be registered in the system? → A: Registry pattern (extensible) - allows future providers without core code changes +- Q: Should users search multiple providers at once? → A: Federated search (merged) - search all enabled providers simultaneously, merge results +- Q: Can book source change after creation? → A: Allow source migration - manual books can be upgraded to external provider books +- Q: How to handle same book from multiple sources? → A: Offer merge on duplicate - prompt user to upgrade manual book or create duplicate +- Q: Where to store provider configuration? → A: Database table (provider_configs) - allows runtime enable/disable +- Q: How many providers to implement initially? → A: Hardcover + OpenLibrary - validates architecture works for multiple providers + +### Session 2026-02-05 (Clarification Round 2) + +- Q: How should provider capabilities be structured and accessed? → A: Static TypeScript interface (IMetadataProvider with boolean flags), checked at runtime via provider registry +- Q: What encryption approach for API keys and credentials? → A: No encryption (plaintext) - acceptable for single-user local SQLite deployments +- Q: What similarity threshold for federated search result deduplication? → A: No deduplication - display all results from all providers without merging +- Q: What should trigger 'degraded' health status? → A: Binary health (healthy/unavailable) - removed 'degraded' state from spec +- Q: Should duplicate detection during migration check other external providers? → A: Single-provider scope - allow same book from multiple external providers +- Q: Circuit breaker cooldown period (inconsistency found)? → A: 60 seconds (resolved inconsistency between Edge Cases and FR-017b) + +### Session 2026-02-05 (Spec Refinement) + +- Q: What events/metrics should be logged for provider operations? → A: Add FR-021 with explicit logging requirements (search requests, metadata fetch, circuit breaker state changes, source migrations, duplicate detection) +- Q: How should federated search results be ranked? → A: Hardcoded priority - Hardcover first, then OpenLibrary (FR-011g) +- Q: What optional metadata fields can users enter for manual books? → A: Add FR-007d - ISBN, publisher, publication date, description, cover image URL (optional fields) +- Q: How should provider configs be initialized? → A: Settings UI approach - all provider configuration managed through application settings (FR-022) +- Q: How should search cache be invalidated? → A: Add FR-011d-1 - invalidate when provider configuration changes +- Q: How to prevent concurrent migration race conditions? → A: Add FR-016f - pessimistic locking for source migration operations + ## User Scenarios & Testing *(mandatory)* ### User Story 1 - Manual Book Addition (Priority: P1) @@ -28,12 +55,12 @@ As a reader with physical books, I want to manually add books to my Tome library **Acceptance Scenarios**: 1. **Given** I'm viewing my library, **When** I click "Add Manual Book" and enter book details (title, author, page count), **Then** the book appears in my library with a visual indicator showing it's a manual entry -5. **Given** I'm filling out the manual book form, **When** I type in a required field (title, author, or page count), **Then** the system validates the field in real-time and displays any validation errors immediately -6. **Given** I've filled out the manual book form with invalid data, **When** I attempt to submit, **Then** the system validates all fields again and prevents submission until all required fields are valid -7. **Given** I'm entering page count in the manual book form, **When** I enter a value less than 1 or greater than 10000, **Then** the system displays a validation error indicating page count must be between 1 and 10000 -4. **Given** I'm adding a manual book with title and author matching an existing book, **When** I submit the form, **Then** system displays a warning about potential duplication but allows me to proceed with creation -2. **Given** I have manually added a book, **When** I log reading progress for that book, **Then** progress is saved and displayed just like Calibre books -3. **Given** I have both Calibre and manual books in my library, **When** I view my library, **Then** both types are displayed together with clear visual differentiation (source badges) +2. **Given** I'm filling out the manual book form, **When** I type in a required field (title, author, or page count), **Then** the system validates the field in real-time and displays any validation errors immediately +3. **Given** I've filled out the manual book form with invalid data, **When** I attempt to submit, **Then** the system validates all fields again and prevents submission until all required fields are valid +4. **Given** I'm entering page count in the manual book form, **When** I enter a value less than 1 or greater than 10000, **Then** the system displays a validation error indicating page count must be between 1 and 10000 +5. **Given** I'm adding a manual book with title and author matching an existing book, **When** I submit the form, **Then** system displays a warning about potential duplication but allows me to proceed with creation +6. **Given** I have manually added a book, **When** I log reading progress for that book, **Then** progress is saved and displayed just like Calibre books +7. **Given** I have both Calibre and manual books in my library, **When** I view my library, **Then** both types are displayed together with clear visual differentiation (source badges) --- @@ -55,7 +82,7 @@ As a user with mixed book sources, I want Calibre syncs to only affect Calibre-s ### User Story 3 - Source-Based Filtering and Display (Priority: P2) -As a user managing multiple book sources, I want to filter my library by source (Calibre vs Manual) so that I can quickly view subsets of my collection based on how I obtained the books. +As a user managing multiple book sources, I want to filter my library by source (Calibre vs Manual vs External Providers) so that I can quickly view subsets of my collection based on how I obtained the books. **Why this priority**: Enhances usability for power users but isn't essential for core functionality. Users can still use the feature effectively without filtering. @@ -65,114 +92,328 @@ As a user managing multiple book sources, I want to filter my library by source 1. **Given** I have books from both Calibre and manual sources, **When** I apply a "Calibre only" filter, **Then** only Calibre-sourced books are displayed 2. **Given** I have books from multiple sources, **When** I apply a "Manual only" filter, **Then** only manually added books are displayed -3. **Given** I have filtered by source, **When** I clear the filter, **Then** all books from all sources are displayed again +3. **Given** I have books from multiple external providers, **When** I apply a "Hardcover" or "OpenLibrary" filter, **Then** only books from that provider are displayed +4. **Given** I have filtered by source, **When** I clear the filter, **Then** all books from all sources are displayed again --- -### User Story 4 - External Metadata Provider Integration (Priority: P3) +### User Story 4 - Federated Metadata Search (Priority: P3) -As a user adding manual books, I want the system to optionally fetch metadata from external providers like Hardcover so that I don't have to manually enter all book details. +As a user adding manual books, I want the system to search multiple external providers simultaneously (Hardcover, OpenLibrary) so that I can find the best metadata match without searching each provider individually. **Why this priority**: Quality-of-life enhancement that reduces friction but isn't required for core functionality. Users can still manually enter all details if needed. -**Independent Test**: Can be tested by initiating a manual book add, searching an external provider, selecting a book, and verifying metadata population. Delivers value through improved user experience. +**Independent Test**: Can be tested by initiating a manual book add, searching for a book title, and verifying results from multiple providers are displayed in a merged list. Delivers value through improved user experience. + +**Acceptance Scenarios**: + +1. **Given** I'm adding a manual book, **When** I search for a book title, **Then** the system searches ALL enabled providers simultaneously within 5 seconds +2. **Given** multiple providers return results, **When** viewing search results, **Then** results are displayed with provider badges (Hardcover, OpenLibrary) and sorted by provider priority (Hardcover first, then OpenLibrary) +3. **Given** I see search results from multiple providers, **When** I select a result, **Then** the book is created with the selected provider's source and metadata +4. **Given** I'm searching for a book, **When** a provider's API request exceeds 5 seconds, **Then** that provider's results are excluded but other providers' results are shown +5. **Given** I'm searching for a book, **When** a provider returns a rate limit error, **Then** that provider's results are excluded with a message that the service is temporarily unavailable +6. **Given** I've selected a book from search results, **When** I review the auto-populated form, **Then** I can edit any field before saving and my edits override the fetched metadata +7. **Given** all enabled providers fail or timeout, **When** the search completes, **Then** the system falls back to the manual entry form with a message explaining provider unavailability + +--- + +### User Story 5 - Source Migration & Duplicate Handling (Priority: P2) + +As a user who has manually added a book, I want the system to detect when I later add the same book from an external provider and offer to upgrade my manual entry, so that I can consolidate my library without duplicates. + +**Why this priority**: Prevents accidental duplicates and allows users to upgrade manual entries with richer metadata when available. Important for long-term library management but not essential for initial functionality. + +**Independent Test**: Can be tested by adding a manual book, then searching for and selecting the same book from Hardcover, and verifying the system offers an upgrade option. Delivers value by reducing duplicate management burden. **Acceptance Scenarios**: -1. **Given** I'm adding a manual book, **When** I search for the book title in the external provider search, **Then** matching books are displayed with cover images and basic metadata -4. **Given** I'm searching for a book in an external provider, **When** the API request exceeds 5 seconds, **Then** the system automatically falls back to the manual entry form and notifies me that external search is unavailable -5. **Given** I'm searching for a book in an external provider, **When** the provider's rate limit is exceeded, **Then** the system automatically falls back to the manual entry form and displays an informative message that the search service is temporarily unavailable -2. **Given** I've searched for a book in an external provider, **When** I select a book from results, **Then** title, author, page count, and cover image are auto-populated in the form -3. **Given** I've auto-populated book details from an external provider, **When** I edit any field before saving, **Then** my manual edits override the fetched metadata +1. **Given** I have a manual book in my library, **When** I add the same book from Hardcover (>85% title+author match), **Then** the system displays a prompt: "This book exists as a manual entry. [Create Duplicate] [Upgrade to Hardcover]" +2. **Given** the system detects a duplicate, **When** I choose "Upgrade to Hardcover", **Then** the existing book's source is changed to 'hardcover', external ID is added, metadata is updated, and all sessions/progress are preserved +3. **Given** the system detects a duplicate, **When** I choose "Create Duplicate", **Then** a new book with source='hardcover' is created and both books exist independently +4. **Given** I have books from two external providers with the same title, **When** adding the book from a third provider, **Then** the system shows all existing sources and offers to create a duplicate +5. **Given** I am upgrading a manual book to an external provider, **When** metadata differs (e.g., page count), **Then** the system shows a confirmation dialog with old vs. new values before applying changes +6. **Given** I have completed a source migration, **When** viewing the book details, **Then** the book shows its new source badge and displays a log entry indicating the migration --- ### Edge Cases -- What happens when a user manually adds a book that already exists in their Calibre library (same title and author)? System checks for title+author matches across all sources, displays a warning showing the existing book(s) and their source(s), but allows the user to proceed with creation after acknowledging the warning. -- How does the system handle manual books when a user later adds the same book to Calibre? Both books remain separate; the user can optionally merge or delete one. -- What happens if an external metadata provider is unavailable or returns no results? System implements a 5-second timeout for API requests. If the provider is unavailable, times out, returns no results, or returns a rate limit error, the system automatically transitions to the manual entry form and displays an informative notification explaining why external search is unavailable. -- How are orphaned Calibre books distinguished from manual books in the UI? Orphaned books have a distinct "orphaned" indicator separate from the source badge. -- What happens when syncing with an empty Calibre library? Manual books remain untouched; only Calibre-sourced books would be orphaned. -- What happens if a user tries to submit the manual book form with missing required fields? Real-time validation displays error messages as the user types in each field. On submission attempt, the system validates all fields again and prevents form submission, highlighting all invalid or empty required fields until corrected. -- What happens if a user enters an invalid page count (e.g., 0, negative, or >10000)? The system displays a real-time validation error indicating that page count must be a positive integer between 1 and 10000, and prevents form submission until corrected. +- **Cross-source duplicates**: When a user manually adds a book that already exists in their Calibre library (same title and author), the system checks for title+author matches across ALL sources, displays a warning showing the existing book(s) and their source(s), and offers to create a duplicate or cancel. +- **Manual to Calibre later addition**: If a user has a manual book and later adds the same book to Calibre, the next Calibre sync creates a new book with source='calibre'. The system SHOULD detect this as a duplicate and offer to migrate the manual book's sessions/progress to the Calibre book (future enhancement, out of scope for Phase 1). +- **Provider unavailability**: If ALL external metadata providers are unavailable, timeout, or return rate limit errors, the system transitions to the pure manual entry form and displays a notification explaining that external search is unavailable. +- **Orphaned Calibre books vs. manual books**: Orphaned books have a distinct "orphaned" indicator separate from the source badge. Only books with source='calibre' can become orphaned. +- **Empty Calibre library sync**: Manual books and external provider books remain untouched; only Calibre-sourced books would be orphaned. +- **Invalid form submission**: Real-time validation displays error messages as the user types. On submission attempt, the system validates all fields again and prevents form submission, highlighting all invalid or empty required fields. +- **Invalid page count**: The system displays a real-time validation error indicating that page count must be a positive integer between 1 and 10000, and prevents form submission until corrected. +- **Provider health degradation**: If a provider fails multiple consecutive requests (5 failures), the system automatically disables that provider via circuit breaker and displays an admin notification. The provider remains disabled for 60 seconds before attempting to re-enable. +- **Source migration rollback**: Source migrations are one-way and cannot be undone. The system logs the migration event for audit purposes but does not support reverting 'hardcover' → 'manual'. ## Requirements *(mandatory)* ### Functional Requirements +#### Core Multi-Source Support + - **FR-001**: System MUST allow Calibre ID to be null for books, enabling books without Calibre sources -- **FR-002**: System MUST track the source of each book (Calibre, Manual, or External Provider like Hardcover) +- **FR-002**: System MUST track the source of each book using a controlled vocabulary: 'calibre', 'manual', 'hardcover', 'openlibrary' +- **FR-002a**: Source MUST be validated against allowed values +- **FR-002b**: Source is generally immutable after creation, except via explicit source migration operations (manual → external provider only) - **FR-003**: System MUST store external provider IDs for books sourced from external metadata providers -- **FR-004**: System MUST restrict sync operations to only affect books where source equals 'calibre' -- **FR-005**: System MUST NOT orphan or remove manual books during Calibre sync operations -- **FR-006**: Users MUST be able to create books manually through a dedicated interface -- **FR-007**: Manual book creation MUST collect at minimum: title, author, and page count -- **FR-007a**: System MUST check for existing books with matching title and author during manual book creation and display a warning to the user, but MUST allow creation to proceed if user confirms -- **FR-007b**: System MUST validate required fields (title, author, page count) in real-time as the user types and perform final validation on form submission, preventing submission if any required field is invalid or empty -- **FR-007c**: System MUST validate page count as a positive integer with a minimum value of 1 and maximum value of 10000, displaying validation errors for values outside this range -- **FR-008**: System MUST display visual indicators (badges/icons) distinguishing book sources in the library -- **FR-009**: System MUST allow all existing Tome features (progress tracking, sessions, goals, streaks) to work identically for manual and Calibre books -- **FR-010**: System MUST provide UI access to add manual books from the main library view -- **FR-011**: System MUST support optional metadata search from external providers during manual book creation -- **FR-011a**: System MUST implement a 5-second timeout for external metadata provider API requests and automatically fallback to manual entry form when timeout is reached or provider is unavailable -- **FR-011b**: System MUST handle external provider rate limit errors by automatically falling back to manual entry form and displaying an informative message to the user that the search service is temporarily unavailable -- **FR-012**: System MUST allow filtering library view by book source (optional for P2) +- **FR-003a**: System MUST enforce uniqueness of (source, externalId) pairs to prevent duplicate provider references +- **FR-003b**: System MUST allow the same book from different sources (different source values create separate book records) +- **FR-004**: System MUST restrict Calibre sync operations to only affect books where source equals 'calibre' +- **FR-005**: System MUST NOT orphan or remove manual books or external provider books during Calibre sync operations -### Key Entities +#### Manual Book Creation -- **Book (Extended)**: Represents any tracked book regardless of source - - Calibre ID (now optional/nullable): Links to Calibre database when source is 'calibre' - - Source: Identifies origin ('calibre', 'manual', 'hardcover') - - External ID: Stores provider-specific ID for books from external services - - Page Count: Positive integer with valid range 1-10000 - - All existing attributes (title, author, status, dates, etc.) remain unchanged +- **FR-006**: Users MUST be able to create books manually through a dedicated interface with source='manual' +- **FR-007**: Manual book creation MUST collect at minimum: title, author, and page count +- **FR-007a**: System MUST check for existing books with matching title and author across ALL sources during manual book creation +- **FR-007b**: System MUST validate required fields (title, author, page count) in real-time as the user types and perform final validation on form submission +- **FR-007c**: System MUST validate page count as a positive integer with a minimum value of 1 and maximum value of 10000 +- **FR-007d**: Manual book creation MAY optionally collect: ISBN, publisher, publication date, description, cover image URL (these fields support richer metadata but are not required) -- **Sync Operation (Behavioral Change)**: Calibre library synchronization - - Only processes books where source equals 'calibre' - - Only orphans books where source equals 'calibre' - - Ignores manual and external provider books completely +#### UI & Display + +- **FR-008**: System MUST display visual indicators (badges/icons) distinguishing book sources in the library (e.g., "Calibre", "Manual", "Hardcover", "OpenLibrary") +- **FR-009**: System MUST allow all existing Tome features (progress tracking, sessions, goals, streaks) to work identically for books from any source +- **FR-010**: System MUST provide UI access to add manual books from the main library view +- **FR-012**: System MUST allow filtering library view by book source with multi-select support (e.g., show only Calibre + Hardcover books) + +#### Provider Architecture + +- **FR-013**: System MUST implement an extensible architecture that supports multiple metadata providers +- **FR-013a**: Each provider MUST implement a TypeScript interface declaring its capabilities via boolean flags (hasSearch, hasMetadataFetch, hasSync, requiresAuth), checked at runtime via the provider registry +- **FR-013b**: Providers MUST be independently enabled or disabled without affecting other providers +- **FR-014**: System MUST maintain provider configurations including enabled status, settings, and credentials +- **FR-014a**: System MUST provide the ability to configure provider-specific settings (timeouts, API endpoints, etc.) +- **FR-014b**: System MUST support both environment variable configuration (for initial setup) and runtime configuration +- **FR-014c**: Credentials MAY be stored in plaintext in provider_configs JSON (acceptable for single-user local SQLite deployments) + +#### Federated Metadata Search + +- **FR-011**: System MUST support federated metadata search from multiple external providers during manual book creation +- **FR-011a**: System MUST search ALL enabled providers simultaneously with a 5-second timeout per provider +- **FR-011b**: System MUST handle provider rate limit errors by excluding that provider from results with an informative message +- **FR-011c**: System MUST display which provider each search result came from (via badges/logos) +- **FR-011d**: System SHOULD cache search results for 5 minutes (TTL) to reduce redundant API calls, keyed by (query, enabled providers) +- **FR-011d-1**: Cache MUST be invalidated when provider configuration changes (enable/disable) +- **FR-011e**: System MUST merge results from multiple providers and display all results without deduplication (users see all provider options) +- **FR-011f**: When all providers fail or timeout, system MUST fall back to pure manual entry form with explanatory message +- **FR-011g**: Search results MUST be sorted by hardcoded provider priority (Hardcover first, then OpenLibrary), preserving each provider's internal result ranking + +#### Source Migration & Duplicate Handling + +- **FR-015**: System MUST detect potential duplicates during book creation using fuzzy matching on title+author (>85% similarity threshold) +- **FR-015a**: When duplicate detected during manual book creation, system MUST show warning but allow user to proceed +- **FR-015b**: When duplicate detected during external provider book creation, system MUST offer: [Create Duplicate] [Upgrade Existing] +- **FR-016**: System MUST support source migration from 'manual' to external provider ('hardcover', 'openlibrary') +- **FR-016a**: Source migration MUST preserve ALL Tome data: sessions, progress, ratings, reviews, shelves +- **FR-016b**: Source migration MUST update book's source, add externalId, and optionally update metadata with user confirmation +- **FR-016c**: Source migration from external provider to external provider is NOT supported (hardcover ↔ openlibrary blocked) +- **FR-016d**: Source migration from external provider back to 'manual' is NOT supported (one-way only) +- **FR-016e**: Duplicate detection during source migration is scoped to the TARGET provider only (e.g., upgrading manual → Hardcover checks only existing Hardcover books, not OpenLibrary books) +- **FR-016f**: Source migration operations MUST use pessimistic locking to prevent concurrent modifications to the same book + +#### Error Handling & Resilience + +- **FR-017**: System MUST implement circuit breaker pattern for each provider independently to prevent cascading failures +- **FR-017a**: Circuit breaker MUST automatically disable provider after 5 consecutive failures +- **FR-017b**: Circuit breaker MUST attempt to re-enable provider after a cooldown period (60 seconds) +- **FR-018**: System MUST implement timeouts for provider operations: + - Search operations: 5 seconds per provider + - Metadata fetch: 10 seconds per provider + - Health checks: 3 seconds per provider +- **FR-019**: System MUST respect provider rate limits and implement appropriate backoff strategies +- **FR-019a**: When provider rate limit reached, system MUST queue requests with exponential backoff or exclude provider from current operation +- **FR-019b**: User MUST see informative message when provider is rate limited or unavailable + +#### Built-in Providers + +- **FR-020**: System MUST include four built-in metadata providers: + - **CalibreProvider**: Syncs existing Calibre books (source='calibre'), always enabled if CALIBRE_DB_PATH set + - **ManualProvider**: User-created books (source='manual'), always enabled, no external metadata + - **HardcoverProvider**: Fetches metadata from Hardcover API (source='hardcover'), requires API key + - **OpenLibraryProvider**: Fetches metadata from OpenLibrary API (source='openlibrary'), no auth required + +#### Observability & Monitoring + +- **FR-021**: System MUST log key provider operations for debugging and monitoring +- **FR-021a**: Provider operations MUST log: search requests (query, provider, result count, duration), metadata fetch requests (provider, externalId, success/failure, duration), circuit breaker state changes (provider, old state, new state, reason) +- **FR-021b**: Source migration operations MUST log: book ID, old source, new source, preserved data summary (session count, progress checkpoints), user ID, timestamp +- **FR-021c**: Duplicate detection events SHOULD log: potential duplicate book IDs, similarity score, user decision (proceed/cancel/upgrade) + +#### Provider Bootstrap & Initialization + +- **FR-022**: Provider configurations MUST be manageable through application settings UI +- **FR-022a**: Settings UI MUST allow users to enable/disable each provider independently +- **FR-022b**: Settings UI MUST allow users to configure provider credentials (API keys) for providers requiring authentication +- **FR-022c**: Settings UI MUST allow users to configure provider-specific settings (timeouts, API endpoints, priority order) + +### Non-Functional Requirements + +- **NFR-001**: Federated search MUST return merged results within 6 seconds (5s provider timeout + 1s processing) +- **NFR-002**: Library filtering by source MUST complete in under 3 seconds for libraries up to 10,000 books +- **NFR-003**: Source migration operation MUST complete in under 2 seconds +- **NFR-004**: Circuit breaker overhead MUST be less than 5ms per provider operation +- **NFR-005**: Provider configuration changes MUST take effect without requiring application restart +- **NFR-006**: System MUST handle up to 4 concurrent external provider API requests without degradation + +### Data Requirements + +#### Book Entity (Schema Extensions) + +**New Fields**: +- `source` (TEXT, NOT NULL, DEFAULT 'calibre'): Identifies book origin - 'calibre', 'manual', 'hardcover', 'openlibrary' +- `externalId` (TEXT, NULLABLE): Provider-specific identifier (e.g., Hardcover book ID, OpenLibrary work ID) + +**Modified Fields**: +- `calibreId` (INTEGER, NULLABLE, previously NOT NULL): Links to Calibre database when source is 'calibre' + +**Constraints**: +- Unique constraint on (source, calibreId) where calibreId is not null +- Unique constraint on (source, externalId) where externalId is not null + +**Migration Requirements**: +- All existing books MUST be migrated to source='calibre' +- Migration MUST preserve all existing data and relationships +- Migration MUST be reversible (rollback capability) + +#### Provider Configuration Entity + +**Purpose**: Store provider-specific settings and credentials + +**Required Fields**: +- `providerId` (TEXT, UNIQUE): Provider identifier +- `enabled` (BOOLEAN): Whether provider is currently active +- `config` (JSON): Provider-specific settings (timeout, API endpoints, rate limits) +- `credentials` (JSON): Authentication credentials (API keys, plaintext storage acceptable for local SQLite) +- `lastHealthCheck` (TIMESTAMP): Last successful health check +- `healthStatus` (TEXT): Current health status - 'healthy' or 'unavailable' (binary state) ## Success Criteria *(mandatory)* ### Measurable Outcomes - **SC-001**: Users can successfully add a manual book and log progress within 2 minutes of first attempt -- **SC-002**: Calibre sync operations complete without affecting manual books (100% isolation verified through testing) -- **SC-003**: Users with mixed book sources report successful tracking of both physical and digital books in post-feature feedback -- **SC-004**: Manual books support all existing Tome features (sessions, progress, goals, streaks) with identical functionality to Calibre books -- **SC-005**: Library view clearly distinguishes book sources with visual indicators that 90%+ of users understand without explanation (based on user testing) -- **SC-006**: Book filtering by source (when implemented) allows users to view source-specific subsets in under 3 seconds -- **SC-007**: External metadata provider integration (when implemented) reduces manual data entry time by 70% for users who use the feature -- **SC-008**: External metadata provider requests that exceed 5 seconds automatically fallback to manual entry without user intervention 100% of the time -- **SC-009**: External metadata provider rate limit errors result in graceful fallback to manual entry with informative messaging 100% of the time +- **SC-002**: Calibre sync operations complete without affecting manual books or external provider books (100% isolation verified through testing) +- **SC-003**: Users with mixed book sources report successful tracking of physical and digital books in post-feature feedback +- **SC-004**: Books from any source support all existing Tome features (sessions, progress, goals, streaks) with identical functionality +- **SC-005**: Library view clearly distinguishes book sources with visual indicators that 90%+ of users understand without explanation +- **SC-006**: Book filtering by source allows users to view source-specific subsets in under 3 seconds +- **SC-007**: Federated metadata search returns merged results from multiple providers within 6 seconds +- **SC-008**: External metadata provider requests that exceed 5 seconds are excluded from results without blocking other providers +- **SC-009**: External metadata provider rate limit errors result in graceful exclusion with informative messaging 100% of the time +- **SC-010**: Source migration from manual to external provider preserves all sessions and progress with 100% fidelity +- **SC-011**: Circuit breaker pattern prevents cascading failures when a provider is down (automatic disable after 5 failures) +- **SC-012**: Provider health monitoring detects and auto-disables unhealthy providers within 30 seconds of repeated failures ## Assumptions -- Users understand the difference between Calibre-sourced and manually added books -- Most users mixing physical and digital books will want to see both in a unified library view by default -- Duplicate books (same title/author from different sources) are acceptable; users can manage duplicates themselves +- Users understand the difference between Calibre-sourced, manually added, and external provider books +- Most users mixing physical and digital books will want to see all sources in a unified library view by default +- Source migration is intentionally one-way (manual → external only) to prevent confusion - Manual books do not require ISBN or other formal identifiers; title/author/pages are sufficient minimums - External metadata providers will use standard REST APIs accessible from the Tome backend -- The Hardcover API (if used) provides adequate metadata (title, author, pages, cover) without authentication requirements +- Hardcover API requires authentication; OpenLibrary API is public and requires no authentication - Sync performance will not degrade significantly with mixed-source libraries of typical size (under 10,000 books) - Visual source indicators (badges/icons) are sufficient differentiation; color-coding is not required -- Users do not expect manual books to retroactively sync with Calibre if later added to Calibre library +- Provider health monitoring and circuit breakers are essential for production stability +- Federated search with 5-second per-provider timeouts provides adequate user experience ## Dependencies - No external service dependencies for core manual book functionality (P1) -- External metadata provider API (e.g., Hardcover) required for P3 auto-population feature -- Existing Tome architecture (database, repositories, sync service) must remain intact +- Hardcover API access required for Hardcover provider (API key required) +- OpenLibrary API access required for OpenLibrary provider (no auth, public API) +- Database schema must support nullable calibreId and new source/externalId fields +- Existing Tome architecture must be extended to support multiple book sources - Calibre database structure remains unchanged (read-only except ratings) ## Out of Scope -- Automatic detection of duplicate books across sources (user must manually identify) -- Migration or merging of manual books into Calibre library +### Phase 1 Out of Scope (May be added in future phases) + +- Automatic session/progress migration between Calibre and manual books when same book detected - Bulk import of manual books from CSV or other formats -- Advanced metadata fields beyond core requirements (ISBN, publisher, publication date optional for future) -- Multi-user access or sharing of manual books +- Multi-provider book linking (one book with multiple external IDs from different providers) +- Advanced metadata fields beyond core requirements for manual books +- Provider plugin system (all providers are built-in for Phase 1) +- Sync orchestration for multiple providers (only Calibre syncs in Phase 1) +- Detailed activity log / audit trail for source migrations +- Admin UI for provider management (API only for Phase 1) +- Advanced duplicate detection (e.g., ISBN matching, cover image similarity) +- Provider marketplace or third-party provider discovery +- Historical provider health metrics and dashboards + +### Permanently Out of Scope + +- Migration or merging of books into Calibre library (Calibre remains source of truth for Calibre books) +- Multi-user access or sharing of manual/external provider books - Export of manual books to other systems (beyond existing Tome export capabilities) -- Editing or updating books in Calibre database from manual entries -- Automatic matching between manual books and later-added Calibre books +- Editing or updating books in Calibre database from manual entries or external providers +- Social features (sharing provider searches, collaborative book lists) +- Automatic background re-syncing of external provider metadata (updates are manual only) + +## Testing Requirements + +### Critical Test Coverage + +- **Sync Isolation**: Calibre sync operations must not touch manual or external provider books +- **Source Filtering**: Library filtering by source must work correctly with mixed sources +- **Federated Search**: Multiple providers must return and merge results correctly +- **Duplicate Detection**: Cross-source duplicate detection must identify matches accurately +- **Source Migration**: Manual to external provider migration must preserve all data +- **Circuit Breaker**: Provider failures must trigger circuit breaker without affecting other providers +- **Timeout Handling**: Provider timeouts must not block federated search operations +- **Rate Limit Handling**: Provider rate limits must be handled gracefully +- **Backward Compatibility**: Existing Calibre-only functionality must work unchanged after schema migration + +### Performance Benchmarks + +- Federated search with 2+ providers: < 6 seconds +- Library filtering by source (10k books): < 3 seconds +- Source migration operation: < 2 seconds +- Circuit breaker overhead: < 5ms per operation + +## Implementation Phases + +This feature should be implemented in logical phases to manage complexity: + +### Phase 1: Foundation (P1 - Manual Books) +- Schema migration (source, externalId, provider configs) +- Manual book creation without external providers +- Sync isolation (Calibre only affects Calibre books) +- Source badges and filtering + +### Phase 2: Provider Infrastructure (P3 - Architecture) +- Provider abstraction and registry +- Provider configuration management +- Circuit breaker and health monitoring +- Error handling and resilience + +### Phase 3: External Providers (P3 - Hardcover + OpenLibrary) +- Hardcover provider implementation +- OpenLibrary provider implementation +- Federated metadata search +- Provider-specific configuration UI/API + +### Phase 4: Advanced Features (P2 - Polish) +- Duplicate detection across sources +- Source migration (manual → external) +- Enhanced search result merging +- Provider health monitoring dashboard + +## Related Documents + +**Architecture Decision Records (to be created)**: +- ADR-015: Multi-Source Provider Architecture +- ADR-016: Book Source Migration Strategy +- ADR-017: Provider Failure Handling with Circuit Breakers +- ADR-018: Calibre ID Nullable Migration + +**Reference Documentation**: +- [Tome Constitution](../../.specify/memory/constitution.md) +- [Tome Patterns](../../.specify/memory/patterns.md) +- [Tome Architecture](../../docs/ARCHITECTURE.md) +- [Repository Pattern Guide](../../docs/REPOSITORY_PATTERN_GUIDE.md) From 99d21147cd929a80bb31716fe1f66b2f9907ac69 Mon Sep 17 00:00:00 2001 From: Mason Fox Date: Thu, 5 Feb 2026 18:16:44 -0500 Subject: [PATCH 03/89] feat: Add implementation plan for multi-source book tracking (spec-003) - Phase 0 research: 10 architectural decisions documented - Provider interface design with capability flags - Federated search with parallel execution - Circuit breaker pattern for resilience - Source migration with transaction safety - Data model with schema changes and migrations Architecture aligns with constitution: - Zero external dependencies (SQLite only) - Repository pattern for all data access - Self-contained deployment - Preserves complete history Next: Generate API contracts and quickstart guide --- specs/003-non-calibre-books/data-model.md | 535 ++++++++++++++++++++++ specs/003-non-calibre-books/plan.md | 151 ++++++ specs/003-non-calibre-books/research.md | 508 ++++++++++++++++++++ 3 files changed, 1194 insertions(+) create mode 100644 specs/003-non-calibre-books/data-model.md create mode 100644 specs/003-non-calibre-books/plan.md create mode 100644 specs/003-non-calibre-books/research.md diff --git a/specs/003-non-calibre-books/data-model.md b/specs/003-non-calibre-books/data-model.md new file mode 100644 index 00000000..161f9736 --- /dev/null +++ b/specs/003-non-calibre-books/data-model.md @@ -0,0 +1,535 @@ +# Data Model: Support Non-Calibre Books + +**Branch**: `003-non-calibre-books` | **Date**: 2026-02-05 + +This document defines the complete data model for multi-source book tracking, including schema changes, new entities, and relationships. + +--- + +## Schema Changes to Existing Entities + +### Books Table (MODIFIED) + +**Purpose**: Support books from multiple sources beyond Calibre + +**New Fields**: +```typescript +{ + source: string; // NEW: 'calibre' | 'manual' | 'hardcover' | 'openlibrary' + externalId: string | null; // NEW: Provider-specific ID (null for manual books) +} +``` + +**Modified Fields**: +```typescript +{ + calibreId: number | null; // CHANGED: Now nullable (was NOT NULL) +} +``` + +**Complete Schema**: +```typescript +// lib/db/schema/books.ts +export const books = sqliteTable('books', { + // Identity + id: integer('id').primaryKey({ autoIncrement: true }), + calibreId: integer('calibre_id'), // NULLABLE (legacy field for Calibre books) + source: text('source', { enum: ['calibre', 'manual', 'hardcover', 'openlibrary'] }) + .notNull() + .default('calibre'), + externalId: text('external_id'), // Provider-specific ID (null for manual) + + // Metadata (existing fields) + title: text('title').notNull(), + authors: text('authors', { mode: 'json' }).$type().notNull(), + isbn: text('isbn'), + totalPages: integer('total_pages'), + publisher: text('publisher'), + publicationDate: text('publication_date'), // YYYY-MM-DD + description: text('description'), + coverImageUrl: text('cover_image_url'), + path: text('path'), // Calibre-specific, null for non-Calibre books + rating: integer('rating'), // 1-5 stars + tags: text('tags', { mode: 'json' }).$type().default([]), + + // Sync metadata + orphaned: integer('orphaned', { mode: 'boolean' }).default(false), + lastSynced: integer('last_synced', { mode: 'timestamp' }), + + // Timestamps + createdAt: integer('created_at', { mode: 'timestamp' }).notNull().default(sql`CURRENT_TIMESTAMP`), + updatedAt: integer('updated_at', { mode: 'timestamp' }).notNull().default(sql`CURRENT_TIMESTAMP`), +}, (table) => ({ + // Legacy Calibre index (keep for backward compatibility) + calibreIdIdx: uniqueIndex('idx_books_calibre_id').on(table.calibreId).where(sql`calibre_id IS NOT NULL`), + + // NEW: Source-specific indexes + sourceIdx: index('idx_books_source').on(table.source), + sourceExternalIdx: uniqueIndex('idx_books_source_external') + .on(table.source, table.externalId) + .where(sql`external_id IS NOT NULL`), +})); +``` + +**Constraints**: +- ✅ Unique on `(source, externalId)` where `externalId IS NOT NULL` +- ✅ Unique on `calibreId` where `calibreId IS NOT NULL` (legacy compatibility) +- ❌ No cross-source duplicate prevention at schema level (handled by application logic) + +**Migration Notes**: +1. **Schema Migration** (`drizzle/00XX_multi_source_support.sql`): + - Add `source` column (default 'calibre') + - Add `externalId` column (nullable) + - Make `calibreId` nullable + - Add new indexes + +2. **Companion Migration** (`lib/migrations/00XX_populate_source_field.ts`): + - Set `source='calibre'` for all existing books + - Validate all existing books have `calibreId` populated + +--- + +## New Entities + +### Provider Configurations Table (NEW) + +**Purpose**: Store runtime configuration for metadata providers + +**Schema**: +```typescript +// lib/db/schema/provider-configs.ts +export const providerConfigs = sqliteTable('provider_configs', { + // Identity + id: integer('id').primaryKey({ autoIncrement: true }), + providerId: text('provider_id').notNull().unique(), // 'hardcover', 'openlibrary' + + // Configuration + enabled: integer('enabled', { mode: 'boolean' }).notNull().default(true), + config: text('config', { mode: 'json' }).$type(), + credentials: text('credentials', { mode: 'json' }).$type(), + + // Health monitoring + lastHealthCheck: integer('last_health_check', { mode: 'timestamp' }), + healthStatus: text('health_status', { enum: ['healthy', 'unavailable'] }) + .notNull() + .default('healthy'), + + // Timestamps + createdAt: integer('created_at', { mode: 'timestamp' }).notNull().default(sql`CURRENT_TIMESTAMP`), + updatedAt: integer('updated_at', { mode: 'timestamp' }).notNull().default(sql`CURRENT_TIMESTAMP`), +}, (table) => ({ + providerIdIdx: uniqueIndex('idx_provider_configs_provider_id').on(table.providerId), +})); + +// Type definitions +export type ProviderConfig = { + timeout?: number; // Request timeout in milliseconds + apiEndpoint?: string; // Override default API endpoint + rateLimit?: { + maxRequests: number; + windowMs: number; + }; +}; + +export type ProviderCredentials = { + apiKey?: string; // Plaintext (acceptable for local deployments) + username?: string; + password?: string; +}; +``` + +**Seeding**: +```typescript +// Default configurations inserted during migration +const defaultConfigs = [ + { + providerId: 'hardcover', + enabled: false, // Requires API key + config: { timeout: 5000 }, + credentials: null, + }, + { + providerId: 'openlibrary', + enabled: true, // Public API, no auth + config: { timeout: 5000 }, + credentials: null, + }, +]; +``` + +**Validation Rules**: +- `providerId` must match existing provider implementations +- `enabled=true` with `requiresAuth=true` requires non-null `credentials.apiKey` +- `config.timeout` must be between 1000-30000ms + +--- + +## Entity Relationships + +### Books ↔ Sources + +``` +Book +├── source: 'calibre' → synced from Calibre DB (read-only except ratings) +├── source: 'manual' → user-created via manual entry form +├── source: 'hardcover' → fetched from Hardcover API +└── source: 'openlibry' → fetched from OpenLibrary API + +Source Identification: +├── calibre: calibreId NOT NULL, externalId NULL +├── manual: calibreId NULL, externalId NULL +├── hardcover: calibreId NULL, externalId = Hardcover book ID +└── openlibrary: calibreId NULL, externalId = OpenLibrary work ID (OLID) +``` + +### Books ↔ Reading Sessions (Existing, No Changes) + +``` +Book (1) ──── (many) ReadingSession + ↓ +Foreign Key: readingSessions.bookId → books.id (CASCADE DELETE) +``` + +**Multi-Source Behavior**: +- Sessions work identically for books from any source +- No schema changes required +- Source badge displayed in UI alongside session metadata + +### Books ↔ Progress Logs (Existing, No Changes) + +``` +Book (1) ──── (many) ProgressLog + ↓ +Foreign Key: progressLogs.bookId → books.id (CASCADE DELETE) +``` + +**Multi-Source Behavior**: +- Progress tracking identical across all sources +- Streaks aggregate progress from ALL sources +- No schema changes required + +--- + +## Data Access Patterns + +### Repository Extensions + +#### BookRepository (EXTENDED) + +**New Methods**: +```typescript +class BookRepository extends BaseRepository { + // Existing methods... + + // NEW: Source-specific queries + async findBySource(source: BookSource): Promise; + async findBySourceAndExternalId(source: BookSource, externalId: string): Promise; + async countBySource(source: BookSource): Promise; + + // EXTENDED: Add source filter parameter + async findWithFilters( + filters: { + status?: string; + search?: string; + tags?: string[]; + rating?: string; + showOrphaned?: boolean; + source?: BookSource | BookSource[]; // NEW: Filter by source(s) + }, + limit: number, + skip: number, + sortBy?: string + ): Promise<{ books: Book[]; total: number }>; + + // EXTENDED: Restrict to Calibre books only + async findNotInCalibreIds( + calibreIds: number[], + filters?: { source: BookSource } // NEW: Default 'calibre' + ): Promise; +} +``` + +#### ProviderConfigRepository (NEW) + +**Methods**: +```typescript +class ProviderConfigRepository extends BaseRepository { + async findByProviderId(providerId: string): Promise; + async findEnabled(): Promise; + async updateHealth(providerId: string, status: ProviderHealth): Promise; + async toggleEnabled(providerId: string, enabled: boolean): Promise; +} +``` + +--- + +## Validation Rules + +### Book Creation + +**Calibre Books** (via sync): +```typescript +{ + source: 'calibre', // Required + calibreId: number, // Required + externalId: null, // Must be null + title: string, // Required + authors: string[], // Required (at least 1) + totalPages: number | null, // Optional +} +``` + +**Manual Books**: +```typescript +{ + source: 'manual', // Required + calibreId: null, // Must be null + externalId: null, // Must be null + title: string, // Required + authors: string[], // Required (at least 1) + totalPages: number, // Required (for progress tracking) + isbn: string | null, // Optional + publisher: string | null, // Optional + publicationDate: string | null, // Optional (YYYY-MM-DD) + description: string | null, // Optional + coverImageUrl: string | null, // Optional (validated URL) +} +``` + +**External Provider Books** (Hardcover, OpenLibrary): +```typescript +{ + source: 'hardcover' | 'openlibrary', // Required + calibreId: null, // Must be null + externalId: string, // Required + title: string, // Required + authors: string[], // Required + // All other fields optional (fetched from provider) +} +``` + +### Provider Configuration + +```typescript +{ + providerId: string, // Must match existing provider + enabled: boolean, // Required + config: { + timeout: number, // 1000-30000ms + apiEndpoint?: string, // Valid URL if provided + }, + credentials: { + apiKey?: string, // Required if provider.requiresAuth=true + } +} +``` + +--- + +## State Transitions + +### Book Source Migration + +**Allowed Transitions**: +``` +manual → hardcover ✅ +manual → openlibrary ✅ +hardcover → manual ❌ (one-way only) +openlibrary → manual ❌ (one-way only) +hardcover → openlibrary ❌ (cross-provider blocked) +calibre → * ❌ (Calibre books cannot be migrated) +``` + +**Migration Process**: +```sql +BEGIN TRANSACTION; + +-- 1. Lock book record +SELECT * FROM books WHERE id = ? FOR UPDATE; + +-- 2. Update source and external ID +UPDATE books +SET source = ?, + external_id = ?, + updated_at = CURRENT_TIMESTAMP +WHERE id = ?; + +-- 3. Optionally update metadata (with user confirmation) +UPDATE books +SET title = ?, authors = ?, total_pages = ?, ... +WHERE id = ?; + +COMMIT; +``` + +**Preserved Data** (automatic via foreign keys): +- Reading sessions (bookId FK) +- Progress logs (bookId FK) +- Book rating (books.rating column) + +--- + +## Index Strategy + +### Performance Targets + +| Query Pattern | Index | Expected Performance | +|--------------|-------|---------------------| +| Filter by source | `idx_books_source` | <50ms for 10k books | +| Find by source+externalId | `idx_books_source_external` | <10ms (unique lookup) | +| Find Calibre book by calibreId | `idx_books_calibre_id` | <10ms (unique lookup) | +| Library with source filter | Composite filter + sort | <3s for 10k books | + +### Index Definitions + +```sql +-- Source filtering (non-unique, many books per source) +CREATE INDEX idx_books_source ON books(source); + +-- External provider lookups (unique, one book per provider+ID) +CREATE UNIQUE INDEX idx_books_source_external + ON books(source, external_id) + WHERE external_id IS NOT NULL; + +-- Legacy Calibre lookups (unique, backward compatibility) +CREATE UNIQUE INDEX idx_books_calibre_id + ON books(calibre_id) + WHERE calibre_id IS NOT NULL; +``` + +--- + +## Data Integrity Rules + +### Constraint Enforcement + +1. **Source-ExternalId Uniqueness**: + - Database enforces via unique index + - Application validates before insert + - Handles duplicate errors gracefully + +2. **Source-Specific Field Requirements**: + ```typescript + if (source === 'calibre') { + assert(calibreId !== null, 'Calibre books require calibreId'); + assert(externalId === null, 'Calibre books must not have externalId'); + } + + if (source === 'manual') { + assert(calibreId === null, 'Manual books must not have calibreId'); + assert(externalId === null, 'Manual books must not have externalId'); + } + + if (['hardcover', 'openlibrary'].includes(source)) { + assert(calibreId === null, 'External books must not have calibreId'); + assert(externalId !== null, 'External books require externalId'); + } + ``` + +3. **Sync Isolation**: + - Calibre sync queries always include `WHERE source = 'calibre'` + - Prevents accidental modification of non-Calibre books + +4. **Migration Validation**: + - Only `source='manual'` books can be migrated + - Target provider must not already have book with same externalId + - Uses pessimistic locking to prevent race conditions + +--- + +## Migration Strategy + +### Phase 1: Schema Changes (Drizzle Migration) + +```sql +-- drizzle/00XX_multi_source_support.sql + +-- 1. Add source column (default 'calibre' for existing books) +ALTER TABLE books ADD COLUMN source TEXT NOT NULL DEFAULT 'calibre'; + +-- 2. Add external_id column (nullable) +ALTER TABLE books ADD COLUMN external_id TEXT; + +-- 3. Make calibre_id nullable (requires table copy in SQLite) +-- (Drizzle generates appropriate ALTER TABLE or table recreation) + +-- 4. Create new indexes +CREATE INDEX idx_books_source ON books(source); +CREATE UNIQUE INDEX idx_books_source_external + ON books(source, external_id) + WHERE external_id IS NOT NULL; + +-- 5. Seed provider_configs table +INSERT INTO provider_configs (provider_id, enabled, config, health_status) +VALUES + ('hardcover', 0, '{"timeout": 5000}', 'healthy'), + ('openlibrary', 1, '{"timeout": 5000}', 'healthy'); +``` + +### Phase 2: Data Migration (Companion Migration) + +```typescript +// lib/migrations/00XX_populate_source_field.ts + +const migration: CompanionMigration = { + name: "00XX_populate_source_field", + requiredTables: ["books"], + description: "Set source='calibre' for all existing books", + + async execute(db) { + // 1. Verify all existing books have calibreId + const booksWithoutCalibreId = db.prepare( + "SELECT id FROM books WHERE calibre_id IS NULL" + ).all(); + + if (booksWithoutCalibreId.length > 0) { + throw new Error(`Found ${booksWithoutCalibreId.length} books without calibreId before migration`); + } + + // 2. Set source='calibre' for all books (already done by DEFAULT) + const result = db.prepare( + "UPDATE books SET source = 'calibre' WHERE source IS NULL" + ).run(); + + logger.info({ updatedCount: result.changes }, "Migration complete"); + } +}; +``` + +### Phase 3: Rollback Plan + +**Rollback SQL**: +```sql +-- Remove new columns (data loss!) +ALTER TABLE books DROP COLUMN source; +ALTER TABLE books DROP COLUMN external_id; + +-- Restore calibre_id NOT NULL constraint +-- (Requires table copy in SQLite) + +-- Drop new table +DROP TABLE provider_configs; +``` + +**Note**: Rollback causes data loss for manual and external provider books. Only use for failed migrations on empty databases. + +--- + +## Summary + +### Schema Impact +- ✅ 1 table modified (`books`) +- ✅ 1 table added (`provider_configs`) +- ✅ 3 indexes added +- ✅ 0 tables dropped +- ✅ Backward compatible (existing Calibre books work unchanged) + +### Repository Impact +- ✅ 1 repository extended (`bookRepository`) +- ✅ 1 repository added (`providerConfigRepository`) +- ✅ Existing repositories unchanged + +### Migration Risk +- 🟡 **Medium**: Requires table copy for nullable calibreId (SQLite limitation) +- ✅ **Mitigated**: Companion migration validates data integrity +- ✅ **Tested**: Fresh database + existing database scenarios diff --git a/specs/003-non-calibre-books/plan.md b/specs/003-non-calibre-books/plan.md new file mode 100644 index 00000000..c2beed08 --- /dev/null +++ b/specs/003-non-calibre-books/plan.md @@ -0,0 +1,151 @@ +# Implementation Plan: Support Non-Calibre Books + +**Branch**: `003-non-calibre-books` | **Date**: 2026-02-05 | **Spec**: [spec.md](./spec.md) +**Input**: Feature specification from `/specs/003-non-calibre-books/spec.md` + +**Note**: This template is filled in by the `/speckit.plan` command. See `.specify/templates/commands/plan.md` for the execution workflow. + +## Summary + +Enable Tome to track books from multiple sources (manual entry, Hardcover, OpenLibrary) while preserving existing Calibre integration. Implements extensible provider architecture with federated search, source migration, and circuit breaker patterns. Foundation enables users to track physical books alongside digital Calibre library. + +## Technical Context + +**Language/Version**: TypeScript 5.x, Node.js 18+ (dev) / Bun 1.x (production) +**Primary Dependencies**: Next.js 16 (App Router), Drizzle ORM, SQLite +**Storage**: SQLite (Tome DB + Calibre DB via factory pattern), JSON for provider configs +**Testing**: Vitest (2000+ existing tests), Repository Pattern with test isolation +**Target Platform**: Self-hosted Linux/Docker (Next.js server on port 3000) +**Project Type**: Web application (Next.js full-stack) +**Performance Goals**: +- Federated search: <6 seconds (5s provider timeout + 1s processing) +- Library filtering: <3 seconds for 10k books +- Source migration: <2 seconds +- Circuit breaker overhead: <5ms per operation + +**Constraints**: +- Zero external service dependencies (must run in complete isolation) +- SQLite only (no Redis, Postgres, cloud APIs) +- Self-contained deployment (Docker single container) +- Calibre integration must remain read-only except ratings/tags +- All existing Tome features must work identically for any book source + +**Scale/Scope**: +- Single-user deployments +- Libraries up to 10k books +- 4 built-in providers initially (Calibre, Manual, Hardcover, OpenLibrary) +- 15+ new API routes, 3+ new repositories, 5+ new services + +## Constitution Check + +*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.* + +Verify compliance with principles from `.specify/memory/constitution.md`: + +- [x] **Data Integrity First**: Schema changes via Drizzle migrations with companion migrations for data transformations. Source field is immutable (except via explicit migration). Uses Database Factory Pattern for all connections. NO writes to Calibre except ratings via `updateCalibreRating()`. +- [x] **Layered Architecture Pattern**: Follows Routes → Services → Repositories. New repositories: `providerRepository`, `providerConfigRepository`. Existing repositories extended for source filtering. NO direct db imports. +- [x] **Self-Contained Deployment**: All providers work in isolation. NO external service dependencies. Hardcover/OpenLibrary APIs are optional (graceful fallback to manual entry). Provider configs stored in SQLite. Uses existing SQLite infrastructure. +- [x] **User Experience Standards**: Smart defaults (source auto-set on creation). Source badges for visual distinction. Duplicate warnings with user choice. Preserves history (migration upgrades manual→external, never deletes). Federated search auto-falls back on failure. +- [x] **Observability & Testing**: Provider operations logged with Pino (search, fetch, circuit breaker state). Tests use real databases with `setDatabase()`. Repository pattern ensures test isolation. New tests for provider architecture, federated search, source migration. + +**Violations**: NONE - Feature aligns with all constitutional principles. + +**Additional Compliance Notes**: +- **Calibre as Source of Truth**: Calibre sync operations restricted to `source='calibre'` books only (FR-004, FR-005) +- **Complete History**: Source migrations are one-way and logged for audit (FR-016d, FR-021b) +- **Zero External Dependencies**: External providers are optional enhancements, not requirements (FR-020) +- **Local-First**: All provider configs stored in SQLite, no cloud dependencies (FR-014c) + +## Project Structure + +### Documentation (this feature) + +```text +specs/[###-feature]/ +├── plan.md # This file (/speckit.plan command output) +├── research.md # Phase 0 output (/speckit.plan command) +├── data-model.md # Phase 1 output (/speckit.plan command) +├── quickstart.md # Phase 1 output (/speckit.plan command) +├── contracts/ # Phase 1 output (/speckit.plan command) +└── tasks.md # Phase 2 output (/speckit.tasks command - NOT created by /speckit.plan) +``` + +### Source Code (repository root) + +```text +# Web application structure (Next.js full-stack) + +# New directories for this feature +lib/ +├── providers/ # NEW: Provider implementations +│ ├── base/ +│ │ ├── IMetadataProvider.ts # Provider interface +│ │ └── ProviderRegistry.ts # Provider discovery/registration +│ ├── calibre.provider.ts # Existing Calibre sync → provider +│ ├── manual.provider.ts # Manual entry (no external API) +│ ├── hardcover.provider.ts # Hardcover API integration +│ └── openlibrary.provider.ts # OpenLibrary API integration +├── services/ +│ ├── provider.service.ts # NEW: Provider orchestration +│ ├── search.service.ts # NEW: Federated search logic +│ ├── migration.service.ts # NEW: Source migration workflows +│ └── circuit-breaker.service.ts # NEW: Failure detection/recovery +├── repositories/ +│ ├── provider-config.repository.ts # NEW: Provider config CRUD +│ └── book.repository.ts # EXTENDED: Source filtering + +# API routes +app/api/ +├── books/ +│ └── route.ts # EXTENDED: Source filtering, manual creation +├── providers/ +│ ├── route.ts # NEW: List providers, health status +│ ├── [providerId]/ +│ │ ├── search/route.ts # NEW: Single provider search +│ │ └── config/route.ts # NEW: Provider config management +│ └── search/route.ts # NEW: Federated search endpoint +└── migration/ + └── [bookId]/route.ts # NEW: Source migration endpoint + +# Database schemas +lib/db/schema/ +├── books.ts # EXTENDED: source, externalId fields +└── provider-configs.ts # NEW: Provider configuration table + +# Frontend components +components/ +├── providers/ +│ ├── ProviderBadge.tsx # NEW: Source indicators +│ ├── FederatedSearchModal.tsx # NEW: Multi-provider search UI +│ ├── SourceMigrationDialog.tsx # NEW: Upgrade manual→external +│ └── DuplicateWarning.tsx # NEW: Cross-source duplicate detection +└── books/ + └── BookFilters.tsx # EXTENDED: Source filter + +# Tests +__tests__/ +├── providers/ +│ ├── provider-registry.test.ts # NEW: Registry tests +│ ├── circuit-breaker.test.ts # NEW: Failure handling +│ └── federated-search.test.ts # NEW: Multi-provider search +├── services/ +│ ├── provider.service.test.ts # NEW: Provider orchestration +│ └── migration.service.test.ts # NEW: Source migration +└── repositories/ + ├── provider-config.repository.test.ts # NEW: Config CRUD + └── book.repository.test.ts # EXTENDED: Source filtering + +# Migrations +drizzle/ +└── 00XX_multi_source_support.sql # NEW: Schema migration (source, externalId) +lib/migrations/ +└── 00XX_populate_source_field.ts # NEW: Companion migration (set source='calibre') +``` + +**Structure Decision**: Follows existing Next.js App Router web application structure with clear separation between provider implementations (`lib/providers/`), business logic (`lib/services/`), and data access (`lib/repositories/`). Uses established Repository Pattern and Service Layer patterns from existing codebase. + +## Complexity Tracking + +> **Fill ONLY if Constitution Check has violations that must be justified** + +**Status**: NOT APPLICABLE - No constitutional violations detected. All complexity is justified by feature requirements and follows established patterns. diff --git a/specs/003-non-calibre-books/research.md b/specs/003-non-calibre-books/research.md new file mode 100644 index 00000000..0ae2323b --- /dev/null +++ b/specs/003-non-calibre-books/research.md @@ -0,0 +1,508 @@ +# Research: Support Non-Calibre Books + +**Branch**: `003-non-calibre-books` | **Date**: 2026-02-05 + +## Phase 0: Architecture Research & Technical Decisions + +This document resolves all "NEEDS CLARIFICATION" items from the implementation plan and establishes architectural patterns for multi-source book tracking. + +--- + +## Research Areas + +### 1. Provider Interface Design + +**Question**: How should the `IMetadataProvider` interface be structured to support diverse provider capabilities? + +**Decision**: Static TypeScript interface with boolean capability flags + +**Rationale**: +- **Extensibility**: New providers can declare capabilities without modifying core code +- **Type safety**: TypeScript enforces interface compliance at compile time +- **Runtime validation**: Registry checks capabilities before dispatching operations +- **Clear contracts**: Explicit boolean flags (hasSearch, hasMetadataFetch, hasSync, requiresAuth) + +**Interface Structure**: +```typescript +export interface IMetadataProvider { + // Identity + id: string; // 'calibre', 'manual', 'hardcover', 'openlibrary' + name: string; // Display name + + // Capabilities (checked at runtime) + capabilities: { + hasSearch: boolean; // Can search for books by query + hasMetadataFetch: boolean; // Can fetch metadata by external ID + hasSync: boolean; // Can sync entire library + requiresAuth: boolean; // Requires API key/credentials + }; + + // Operations (optional - checked via capabilities) + search?(query: string): Promise; + fetchMetadata?(externalId: string): Promise; + sync?(): Promise; + + // Health monitoring + healthCheck(): Promise; +} + +export type ProviderHealth = 'healthy' | 'unavailable'; +``` + +**Alternatives Considered**: +- ❌ **Abstract base class with inheritance**: Too rigid, forces providers to inherit methods they don't use +- ❌ **Plugin system with dynamic loading**: Over-engineered for 4 built-in providers +- ❌ **Separate interfaces per capability**: Verbose, harder to reason about provider capabilities at a glance + +**Implementation Notes**: +- Registry validates capabilities match implemented methods at startup +- Missing optional methods throw descriptive errors if called despite capability flags +- Health checks run independently per provider (circuit breaker pattern) + +--- + +### 2. Provider Configuration Storage + +**Question**: Where and how should provider configurations be stored? + +**Decision**: SQLite `provider_configs` table with JSON fields for settings/credentials + +**Rationale**: +- **Consistency**: Uses existing SQLite infrastructure (no new storage layer) +- **Self-contained**: Aligns with constitution's zero external dependencies principle +- **Flexibility**: JSON fields allow provider-specific settings without schema changes +- **Simplicity**: Plaintext credentials acceptable for single-user local deployments + +**Schema Design**: +```sql +CREATE TABLE provider_configs ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + provider_id TEXT UNIQUE NOT NULL, -- 'hardcover', 'openlibrary' + enabled BOOLEAN NOT NULL DEFAULT 1, -- Runtime enable/disable + config JSON, -- { timeout: 5000, apiEndpoint: '...' } + credentials JSON, -- { apiKey: '...' } (plaintext) + last_health_check TIMESTAMP, + health_status TEXT, -- 'healthy' | 'unavailable' + created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, + updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP +); +``` + +**Alternatives Considered**: +- ❌ **Environment variables only**: No runtime enable/disable, requires restart for config changes +- ❌ **Encrypted credentials**: Over-engineered for single-user deployments (see FR-014c) +- ❌ **Separate tables per provider**: Schema complexity, harder to manage uniformly + +**Migration Strategy**: +1. Create `provider_configs` table via Drizzle migration +2. Seed with default configs for built-in providers +3. Calibre provider config only if `CALIBRE_DB_PATH` env var set + +--- + +### 3. Federated Search Architecture + +**Question**: How should federated search coordinate multiple providers efficiently? + +**Decision**: Parallel promise-based dispatch with provider-level timeouts and result merging + +**Rationale**: +- **Performance**: Parallel execution prevents slow providers from blocking fast ones +- **Resilience**: Provider-level timeouts (5s) and error handling isolate failures +- **User experience**: Partial results better than all-or-nothing (graceful degradation) +- **Simplicity**: Promise.allSettled() handles race conditions without complex orchestration + +**Flow Architecture**: +``` +User Query + ↓ +SearchService.federatedSearch(query) + ↓ +Promise.allSettled([ + hardcoverProvider.search(query) → 5s timeout → [results] or error, + openlibraryProvider.search(query) → 5s timeout → [results] or error +]) + ↓ +Filter fulfilled promises → Merge results → Sort by priority + ↓ +Return { results: BookMetadata[], errors: ProviderError[] } +``` + +**Caching Strategy** (FR-011d): +- **Key**: `${query}:${enabledProviderIds.sort().join(',')}` +- **TTL**: 5 minutes +- **Invalidation**: Provider enable/disable triggers cache clear +- **Storage**: In-memory Map (singleton service instance) + +**Priority Sorting** (FR-011g): +- Hardcoded order: Hardcover → OpenLibrary +- Within provider: Preserve API's internal ranking +- No cross-provider deduplication (FR-015b) + +**Alternatives Considered**: +- ❌ **Sequential search with early exit**: Slower, defeats parallelism purpose +- ❌ **Redis cache**: External dependency violates constitution +- ❌ **Persistent cache in SQLite**: Stale data issues, cache invalidation complexity + +--- + +### 4. Circuit Breaker Pattern + +**Question**: How should the circuit breaker pattern prevent cascading failures? + +**Decision**: Per-provider state machine with automatic disable/re-enable + +**Rationale**: +- **Isolation**: Provider failures don't cascade to other providers or core features +- **Automatic recovery**: Cooldown period (60s) allows self-healing without manual intervention +- **Observability**: State changes logged for debugging (FR-021a) +- **Simplicity**: Three states (CLOSED, OPEN, HALF_OPEN) with clear transitions + +**State Machine**: +``` +CLOSED (healthy) + ↓ (5 consecutive failures) +OPEN (disabled) + ↓ (60s cooldown) +HALF_OPEN (testing) + ↓ (1 success) ↓ (1 failure) +CLOSED OPEN +``` + +**Implementation**: +```typescript +class CircuitBreaker { + private state: 'CLOSED' | 'OPEN' | 'HALF_OPEN' = 'CLOSED'; + private failureCount = 0; + private lastFailureTime: Date | null = null; + + async execute(operation: () => Promise): Promise { + if (this.state === 'OPEN') { + if (Date.now() - this.lastFailureTime > 60000) { + this.state = 'HALF_OPEN'; + } else { + throw new Error('Circuit breaker OPEN'); + } + } + + try { + const result = await operation(); + if (this.state === 'HALF_OPEN') { + this.state = 'CLOSED'; + this.failureCount = 0; + } + return result; + } catch (error) { + this.failureCount++; + this.lastFailureTime = new Date(); + + if (this.failureCount >= 5) { + this.state = 'OPEN'; + logger.error({ provider: this.providerId }, 'Circuit breaker OPEN'); + } + + throw error; + } + } +} +``` + +**Alternatives Considered**: +- ❌ **Exponential backoff without breaker**: Accumulates retries, wastes resources +- ❌ **Manual admin intervention required**: Violates "make complexity invisible" principle +- ❌ **Permanent disable**: User loses functionality until manual re-enable + +--- + +### 5. Duplicate Detection Algorithm + +**Question**: What similarity algorithm should detect duplicate books across sources? + +**Decision**: Levenshtein distance with >85% threshold on normalized title+author + +**Rationale**: +- **Simple and proven**: Standard fuzzy matching algorithm +- **Configurable**: Threshold can be tuned based on user feedback +- **Efficient**: O(n*m) acceptable for small candidate sets (<100 books per query) +- **Good enough**: Catches typos, punctuation differences, minor variations + +**Normalization**: +```typescript +function normalize(text: string): string { + return text + .toLowerCase() + .replace(/[^\w\s]/g, '') // Remove punctuation + .replace(/\s+/g, ' ') // Normalize whitespace + .trim(); +} + +function similarity(a: string, b: string): number { + const dist = levenshtein(normalize(a), normalize(b)); + const maxLen = Math.max(a.length, b.length); + return (1 - dist / maxLen) * 100; +} +``` + +**Matching Strategy** (FR-015a, FR-015b): +- **Manual entry**: Check all existing books, show warning, allow proceed +- **External provider**: Check target provider only (FR-016e), offer upgrade/duplicate +- **Threshold**: >85% similarity on `title + ' ' + author` + +**Alternatives Considered**: +- ❌ **ISBN matching**: Many books lack ISBNs (especially physical books) +- ❌ **Cover image similarity**: Computationally expensive, unreliable for different editions +- ❌ **Metadata hash**: Too strict, misses legitimate duplicates with minor differences + +--- + +### 6. Source Migration Data Preservation + +**Question**: How should source migration preserve all Tome data during manual→external upgrades? + +**Decision**: Transactional update with foreign key cascades and pessimistic locking + +**Rationale**: +- **Data integrity**: Transaction ensures all-or-nothing migration +- **Referential integrity**: Foreign key cascades automatically update related records +- **Concurrency safety**: Pessimistic lock prevents race conditions (FR-016f) +- **Audit trail**: Migration logged for debugging (FR-021b) + +**Migration Process**: +```typescript +async migrateSource(bookId: number, newSource: string, externalId: string) { + return db.transaction(async () => { + // 1. Lock book record + const book = await db.select() + .from(books) + .where(eq(books.id, bookId)) + .for('UPDATE') // Pessimistic lock + .get(); + + // 2. Validate migration rules + if (book.source !== 'manual') { + throw new Error('Only manual books can be migrated'); + } + + // 3. Check for duplicate in target provider + const duplicate = await bookRepository.findBySourceAndExternalId(newSource, externalId); + if (duplicate) { + throw new Error('Book already exists in target provider'); + } + + // 4. Update book record (sessions, progress, ratings cascade automatically) + await bookRepository.update(bookId, { + source: newSource, + externalId: externalId, + // Optionally update metadata with user confirmation + }); + + // 5. Log migration event + logger.info({ + bookId, + oldSource: 'manual', + newSource, + externalId, + sessionCount: await sessionRepository.countByBookId(bookId), + progressCount: await progressRepository.countByBookId(bookId), + }, 'Source migration completed'); + }); +} +``` + +**Alternatives Considered**: +- ❌ **Create new book + move references**: Complex, error-prone, disrupts foreign keys +- ❌ **Optimistic locking with retry**: Race conditions possible with concurrent UI operations +- ❌ **Soft delete old + create new**: Violates "preserve complete history" principle + +--- + +### 7. Calibre Sync Isolation + +**Question**: How should Calibre sync operations be restricted to only affect Calibre-sourced books? + +**Decision**: Add `source='calibre'` filter to all sync queries + +**Rationale**: +- **Surgical change**: Minimal modification to existing sync logic +- **Constitution compliance**: Enforces "Calibre as source of truth" for Calibre books only +- **Data safety**: Manual and external provider books immune to Calibre sync operations +- **Backward compatible**: Existing books migrated to `source='calibre'` preserve behavior + +**Implementation Changes**: +```typescript +// lib/sync-service.ts + +// BEFORE: +const orphanedBooks = await bookRepository.findNotInCalibreIds(calibreIds); + +// AFTER: +const orphanedBooks = await bookRepository.findNotInCalibreIds(calibreIds, { + source: 'calibre' // Only mark Calibre-sourced books as orphaned +}); + +// BEFORE: +const existingBook = await bookRepository.findByCalibreId(calibreBook.id); + +// AFTER: +const existingBook = await bookRepository.findByCalibreId(calibreBook.id, { + source: 'calibre' // Only update Calibre-sourced books +}); +``` + +**Alternatives Considered**: +- ❌ **Separate sync services per source**: Code duplication, harder to maintain +- ❌ **Source-specific repositories**: Over-engineered for source filtering use case + +--- + +### 8. Provider Health Monitoring + +**Question**: What metrics should trigger provider health state changes? + +**Decision**: Binary health states (healthy/unavailable) with circuit breaker integration + +**Rationale**: +- **Simplicity**: Binary states clearer than degraded/partial states (per clarification Q33) +- **Automatic detection**: Circuit breaker state machine handles failures without separate monitoring +- **Lightweight**: Health checks piggybacked on normal operations (no separate polling) +- **User-visible**: Health status exposed via API for settings UI + +**Health Check Flow**: +``` +Provider Operation Attempt + ↓ +Circuit Breaker.execute() + ↓ (success) ↓ (failure) +Update health Increment failure count +status='healthy' Check if threshold reached (5) + ↓ ↓ +Continue Update health status='unavailable' +``` + +**Alternatives Considered**: +- ❌ **Scheduled health checks**: Adds complexity, wastes API calls for unused providers +- ❌ **Degraded state**: User confusion about what "degraded" means operationally +- ❌ **Response time monitoring**: Over-engineered for single-user deployments + +--- + +### 9. Provider Priority Configuration + +**Question**: Should provider priority be user-configurable or hardcoded? + +**Decision**: Hardcoded priority for Phase 1 (Hardcover → OpenLibrary) + +**Rationale**: +- **Simplicity**: Avoids UI complexity for marginal benefit +- **Sufficient**: Most users don't care about search result ordering +- **Future enhancement**: Can add priority field to provider_configs if users request it +- **Clear default**: Hardcover (curated, higher quality) before OpenLibrary (comprehensive) + +**Implementation**: +```typescript +const PROVIDER_PRIORITY: Record = { + hardcover: 1, + openlibrary: 2, +}; + +function sortResults(results: SearchResult[]): SearchResult[] { + return results.sort((a, b) => { + const priorityDiff = PROVIDER_PRIORITY[a.source] - PROVIDER_PRIORITY[b.source]; + if (priorityDiff !== 0) return priorityDiff; + return a.index - b.index; // Preserve provider's internal order + }); +} +``` + +**Alternatives Considered**: +- ❌ **User-configurable priority**: Adds UI complexity, most users won't use it +- ❌ **Machine learning ranking**: Massive over-engineering for book search + +--- + +### 10. External API Integration Patterns + +**Question**: How should external API integrations (Hardcover, OpenLibrary) be structured? + +**Decision**: Provider-specific service classes with retry logic and rate limit handling + +**Rationale**: +- **Encapsulation**: API-specific logic isolated in provider implementations +- **Resilience**: Retry with exponential backoff handles transient failures +- **Rate limiting**: Providers detect rate limit responses and trigger circuit breaker +- **Testability**: Mock HTTP responses without affecting core logic + +**Hardcover Provider Structure**: +```typescript +export class HardcoverProvider implements IMetadataProvider { + id = 'hardcover'; + name = 'Hardcover'; + capabilities = { + hasSearch: true, + hasMetadataFetch: true, + hasSync: false, + requiresAuth: true, + }; + + private apiKey: string; + private baseUrl = 'https://api.hardcover.app/v1'; + + async search(query: string): Promise { + const response = await this.retryRequest(async () => { + return fetch(`${this.baseUrl}/books/search?q=${encodeURIComponent(query)}`, { + headers: { 'Authorization': `Bearer ${this.apiKey}` }, + signal: AbortSignal.timeout(5000), // 5s timeout + }); + }); + + if (response.status === 429) { + throw new RateLimitError('Hardcover API rate limit exceeded'); + } + + return this.parseSearchResults(await response.json()); + } + + private async retryRequest(fn: () => Promise, retries = 3): Promise { + // Exponential backoff retry logic + } +} +``` + +**OpenLibrary Provider Structure**: +- Public API (no auth required) +- Base URL: `https://openlibrary.org/` +- Search endpoint: `/search.json?q=...` +- Works endpoint: `/works/{OLID}.json` + +**Alternatives Considered**: +- ❌ **Shared API client**: Providers have different auth/headers, hard to generalize +- ❌ **No retry logic**: Transient network failures break user experience + +--- + +## Summary of Architectural Decisions + +| Area | Decision | Key Benefit | +|------|----------|-------------| +| Provider Interface | TypeScript interface with capability flags | Type safety + runtime flexibility | +| Config Storage | SQLite JSON fields | Self-contained, no external deps | +| Federated Search | Parallel Promise.allSettled | Performance + resilience | +| Circuit Breaker | Per-provider state machine | Automatic failure isolation | +| Duplicate Detection | Levenshtein >85% threshold | Simple, proven, configurable | +| Source Migration | Transactional updates with locking | Data integrity + concurrency safety | +| Calibre Sync | Source filter in queries | Surgical change, preserves isolation | +| Health Monitoring | Binary states + circuit breaker | Simplicity, automatic detection | +| Provider Priority | Hardcoded for Phase 1 | Avoid premature optimization | +| API Integration | Provider-specific services | Encapsulation + testability | + +--- + +## Next Steps + +Phase 0 complete. All architectural unknowns resolved. Ready to proceed to Phase 1 (Design & Contracts). + +**Phase 1 will generate**: +1. `data-model.md` - Complete entity schemas with relationships +2. `contracts/` - OpenAPI specs for new API routes +3. `quickstart.md` - Developer onboarding guide for provider system +4. Agent context updates - Technology additions for AI assistants From 33d95af4190c3476f20e7ff5698af51cf1c96ef9 Mon Sep 17 00:00:00 2001 From: Mason Fox Date: Thu, 5 Feb 2026 18:23:11 -0500 Subject: [PATCH 04/89] feat(spec-003): Phase 1 - Database schema and provider foundation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Extended books schema with source/externalId fields for multi-source support - Made calibreId, path, lastSynced nullable (non-Calibre books) - Created provider_configs table for provider runtime configuration - Generated Drizzle migration 0022 with data preservation - Created companion migration to seed default provider configs - Implemented IMetadataProvider interface with capability flags - Implemented ProviderRegistry for provider management - Fixed existing code to handle nullable calibreId fields - Updated rating sync to only affect Calibre books Phase 1 Complete: ✅ T001-T008: Database schema, migrations, provider interface ✅ All TypeScript errors resolved ✅ Backward compatible with existing Calibre books Next: Phase 2 - Repository extensions and services --- drizzle/0022_nappy_spectrum.sql | 56 + drizzle/meta/0022_snapshot.json | 1074 ++++++++++++++++++ drizzle/meta/_journal.json | 7 + lib/db/schema/books.ts | 24 +- lib/db/schema/index.ts | 1 + lib/db/schema/provider-configs.ts | 70 ++ lib/migrations/0022_seed_provider_configs.ts | 147 +++ lib/providers/base/IMetadataProvider.ts | 197 ++++ lib/providers/base/ProviderRegistry.ts | 282 +++++ lib/repositories/book.repository.ts | 8 +- lib/services/session.service.ts | 8 +- 11 files changed, 1865 insertions(+), 9 deletions(-) create mode 100644 drizzle/0022_nappy_spectrum.sql create mode 100644 drizzle/meta/0022_snapshot.json create mode 100644 lib/db/schema/provider-configs.ts create mode 100644 lib/migrations/0022_seed_provider_configs.ts create mode 100644 lib/providers/base/IMetadataProvider.ts create mode 100644 lib/providers/base/ProviderRegistry.ts diff --git a/drizzle/0022_nappy_spectrum.sql b/drizzle/0022_nappy_spectrum.sql new file mode 100644 index 00000000..add39be5 --- /dev/null +++ b/drizzle/0022_nappy_spectrum.sql @@ -0,0 +1,56 @@ +CREATE TABLE `provider_configs` ( + `id` integer PRIMARY KEY AUTOINCREMENT NOT NULL, + `provider` text NOT NULL, + `display_name` text NOT NULL, + `enabled` integer DEFAULT true NOT NULL, + `capabilities` text NOT NULL, + `settings` text DEFAULT '{}' NOT NULL, + `credentials` text DEFAULT '{}', + `circuit_state` text DEFAULT 'CLOSED' NOT NULL, + `last_failure` integer, + `failure_count` integer DEFAULT 0 NOT NULL, + `health_status` text DEFAULT 'healthy' NOT NULL, + `last_health_check` integer, + `priority` integer DEFAULT 100 NOT NULL, + `created_at` integer DEFAULT (unixepoch()) NOT NULL, + `updated_at` integer DEFAULT (unixepoch()) NOT NULL +); +--> statement-breakpoint +CREATE UNIQUE INDEX `provider_configs_provider_unique` ON `provider_configs` (`provider`);--> statement-breakpoint +CREATE INDEX `idx_provider_configs_enabled` ON `provider_configs` (`enabled`);--> statement-breakpoint +CREATE INDEX `idx_provider_configs_priority` ON `provider_configs` (`priority`);--> statement-breakpoint +PRAGMA foreign_keys=OFF;--> statement-breakpoint +CREATE TABLE `__new_books` ( + `id` integer PRIMARY KEY AUTOINCREMENT NOT NULL, + `calibre_id` integer, + `source` text DEFAULT 'calibre' NOT NULL, + `external_id` text, + `title` text NOT NULL, + `authors` text DEFAULT '[]' NOT NULL, + `author_sort` text, + `isbn` text, + `total_pages` integer, + `added_to_library` integer DEFAULT (unixepoch()) NOT NULL, + `last_synced` integer, + `publisher` text, + `pub_date` integer, + `series` text, + `series_index` real, + `tags` text DEFAULT '[]' NOT NULL, + `path` text, + `description` text, + `rating` integer, + `orphaned` integer DEFAULT false NOT NULL, + `orphaned_at` integer, + `created_at` integer DEFAULT (unixepoch()) NOT NULL, + `updated_at` integer DEFAULT (unixepoch()) NOT NULL +); +--> statement-breakpoint +INSERT INTO `__new_books`("id", "calibre_id", "source", "external_id", "title", "authors", "author_sort", "isbn", "total_pages", "added_to_library", "last_synced", "publisher", "pub_date", "series", "series_index", "tags", "path", "description", "rating", "orphaned", "orphaned_at", "created_at", "updated_at") SELECT "id", "calibre_id", 'calibre', CAST("calibre_id" AS TEXT), "title", "authors", "author_sort", "isbn", "total_pages", "added_to_library", "last_synced", "publisher", "pub_date", "series", "series_index", "tags", "path", "description", "rating", "orphaned", "orphaned_at", "created_at", "updated_at" FROM `books`;--> statement-breakpoint +DROP TABLE `books`;--> statement-breakpoint +ALTER TABLE `__new_books` RENAME TO `books`;--> statement-breakpoint +PRAGMA foreign_keys=ON;--> statement-breakpoint +CREATE UNIQUE INDEX `books_calibre_id_unique` ON `books` (`calibre_id`);--> statement-breakpoint +CREATE INDEX `idx_books_author_sort` ON `books` (`author_sort`);--> statement-breakpoint +CREATE INDEX `idx_books_source` ON `books` (`source`);--> statement-breakpoint +CREATE UNIQUE INDEX `idx_books_source_external` ON `books` (`source`,`external_id`) WHERE external_id IS NOT NULL; \ No newline at end of file diff --git a/drizzle/meta/0022_snapshot.json b/drizzle/meta/0022_snapshot.json new file mode 100644 index 00000000..ed73d532 --- /dev/null +++ b/drizzle/meta/0022_snapshot.json @@ -0,0 +1,1074 @@ +{ + "version": "6", + "dialect": "sqlite", + "id": "938bdd32-fb3a-4347-b852-c3d118bc7508", + "prevId": "bd8662aa-ef84-4ee3-b31a-ee57e29cbf9e", + "tables": { + "books": { + "name": "books", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "calibre_id": { + "name": "calibre_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "source": { + "name": "source", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'calibre'" + }, + "external_id": { + "name": "external_id", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "title": { + "name": "title", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "authors": { + "name": "authors", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'[]'" + }, + "author_sort": { + "name": "author_sort", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "isbn": { + "name": "isbn", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "total_pages": { + "name": "total_pages", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "added_to_library": { + "name": "added_to_library", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "last_synced": { + "name": "last_synced", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "publisher": { + "name": "publisher", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "pub_date": { + "name": "pub_date", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "series": { + "name": "series", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "series_index": { + "name": "series_index", + "type": "real", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "tags": { + "name": "tags", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'[]'" + }, + "path": { + "name": "path", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "description": { + "name": "description", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "rating": { + "name": "rating", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "orphaned": { + "name": "orphaned", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "orphaned_at": { + "name": "orphaned_at", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "books_calibre_id_unique": { + "name": "books_calibre_id_unique", + "columns": [ + "calibre_id" + ], + "isUnique": true + }, + "idx_books_author_sort": { + "name": "idx_books_author_sort", + "columns": [ + "author_sort" + ], + "isUnique": false + }, + "idx_books_source": { + "name": "idx_books_source", + "columns": [ + "source" + ], + "isUnique": false + }, + "idx_books_source_external": { + "name": "idx_books_source_external", + "columns": [ + "source", + "external_id" + ], + "isUnique": true, + "where": "external_id IS NOT NULL" + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "reading_sessions": { + "name": "reading_sessions", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "user_id": { + "name": "user_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "book_id": { + "name": "book_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "session_number": { + "name": "session_number", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'to-read'" + }, + "started_date": { + "name": "started_date", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "completed_date": { + "name": "completed_date", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "review": { + "name": "review", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "is_active": { + "name": "is_active", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "read_next_order": { + "name": "read_next_order", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "idx_book_session": { + "name": "idx_book_session", + "columns": [ + "book_id", + "session_number" + ], + "isUnique": true + }, + "idx_active_session": { + "name": "idx_active_session", + "columns": [ + "book_id" + ], + "isUnique": true, + "where": "\"reading_sessions\".\"is_active\" = 1" + }, + "idx_sessions_book_id": { + "name": "idx_sessions_book_id", + "columns": [ + "book_id" + ], + "isUnique": false + }, + "idx_sessions_status": { + "name": "idx_sessions_status", + "columns": [ + "status" + ], + "isUnique": false + }, + "idx_sessions_user_book": { + "name": "idx_sessions_user_book", + "columns": [ + "user_id", + "book_id" + ], + "isUnique": false + }, + "idx_sessions_read_next_order": { + "name": "idx_sessions_read_next_order", + "columns": [ + "read_next_order", + "id" + ], + "isUnique": false, + "where": "\"reading_sessions\".\"status\" = 'read-next'" + } + }, + "foreignKeys": { + "reading_sessions_book_id_books_id_fk": { + "name": "reading_sessions_book_id_books_id_fk", + "tableFrom": "reading_sessions", + "tableTo": "books", + "columnsFrom": [ + "book_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "progress_logs": { + "name": "progress_logs", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "user_id": { + "name": "user_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "book_id": { + "name": "book_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "session_id": { + "name": "session_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "current_page": { + "name": "current_page", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "current_percentage": { + "name": "current_percentage", + "type": "real", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "progress_date": { + "name": "progress_date", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "notes": { + "name": "notes", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "pages_read": { + "name": "pages_read", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "idx_progress_book_date": { + "name": "idx_progress_book_date", + "columns": [ + "book_id", + "progress_date" + ], + "isUnique": false + }, + "idx_progress_session_date": { + "name": "idx_progress_session_date", + "columns": [ + "session_id", + "progress_date" + ], + "isUnique": false + }, + "idx_progress_user_date": { + "name": "idx_progress_user_date", + "columns": [ + "user_id", + "progress_date" + ], + "isUnique": false + }, + "idx_progress_date": { + "name": "idx_progress_date", + "columns": [ + "progress_date" + ], + "isUnique": false + } + }, + "foreignKeys": { + "progress_logs_book_id_books_id_fk": { + "name": "progress_logs_book_id_books_id_fk", + "tableFrom": "progress_logs", + "tableTo": "books", + "columnsFrom": [ + "book_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "progress_logs_session_id_reading_sessions_id_fk": { + "name": "progress_logs_session_id_reading_sessions_id_fk", + "tableFrom": "progress_logs", + "tableTo": "reading_sessions", + "columnsFrom": [ + "session_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": { + "current_page_check": { + "name": "current_page_check", + "value": "\"progress_logs\".\"current_page\" >= 0" + }, + "current_percentage_check": { + "name": "current_percentage_check", + "value": "\"progress_logs\".\"current_percentage\" >= 0 AND \"progress_logs\".\"current_percentage\" <= 100" + }, + "pages_read_check": { + "name": "pages_read_check", + "value": "\"progress_logs\".\"pages_read\" >= 0" + } + } + }, + "streaks": { + "name": "streaks", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "user_id": { + "name": "user_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "streak_enabled": { + "name": "streak_enabled", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": false + }, + "current_streak": { + "name": "current_streak", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "longest_streak": { + "name": "longest_streak", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "last_activity_date": { + "name": "last_activity_date", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "streak_start_date": { + "name": "streak_start_date", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "total_days_active": { + "name": "total_days_active", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "daily_threshold": { + "name": "daily_threshold", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 1 + }, + "user_timezone": { + "name": "user_timezone", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'America/New_York'" + }, + "last_checked_date": { + "name": "last_checked_date", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "idx_streak_user": { + "name": "idx_streak_user", + "columns": [ + "COALESCE(\"user_id\", -1)" + ], + "isUnique": true + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "reading_goals": { + "name": "reading_goals", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "user_id": { + "name": "user_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "year": { + "name": "year", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "books_goal": { + "name": "books_goal", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "idx_goal_user_year": { + "name": "idx_goal_user_year", + "columns": [ + "COALESCE(\"user_id\", -1)", + "year" + ], + "isUnique": true + }, + "idx_goal_year": { + "name": "idx_goal_year", + "columns": [ + "year" + ], + "isUnique": false + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": { + "books_goal_check": { + "name": "books_goal_check", + "value": "\"reading_goals\".\"books_goal\" >= 1" + }, + "year_range_check": { + "name": "year_range_check", + "value": "\"reading_goals\".\"year\" >= 1900 AND \"reading_goals\".\"year\" <= 9999" + } + } + }, + "book_shelves": { + "name": "book_shelves", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "shelf_id": { + "name": "shelf_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "book_id": { + "name": "book_id", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "sort_order": { + "name": "sort_order", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "added_at": { + "name": "added_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "idx_book_shelves_shelf": { + "name": "idx_book_shelves_shelf", + "columns": [ + "shelf_id" + ], + "isUnique": false + }, + "idx_book_shelves_book": { + "name": "idx_book_shelves_book", + "columns": [ + "book_id" + ], + "isUnique": false + }, + "idx_book_shelves_unique": { + "name": "idx_book_shelves_unique", + "columns": [ + "shelf_id", + "book_id" + ], + "isUnique": true + }, + "idx_book_shelves_shelf_order": { + "name": "idx_book_shelves_shelf_order", + "columns": [ + "shelf_id", + "sort_order" + ], + "isUnique": false + } + }, + "foreignKeys": { + "book_shelves_shelf_id_shelves_id_fk": { + "name": "book_shelves_shelf_id_shelves_id_fk", + "tableFrom": "book_shelves", + "tableTo": "shelves", + "columnsFrom": [ + "shelf_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + }, + "book_shelves_book_id_books_id_fk": { + "name": "book_shelves_book_id_books_id_fk", + "tableFrom": "book_shelves", + "tableTo": "books", + "columnsFrom": [ + "book_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "shelves": { + "name": "shelves", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "user_id": { + "name": "user_id", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "name": { + "name": "name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "description": { + "name": "description", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "color": { + "name": "color", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "icon": { + "name": "icon", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "idx_shelves_user": { + "name": "idx_shelves_user", + "columns": [ + "user_id" + ], + "isUnique": false + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + }, + "provider_configs": { + "name": "provider_configs", + "columns": { + "id": { + "name": "id", + "type": "integer", + "primaryKey": true, + "notNull": true, + "autoincrement": true + }, + "provider": { + "name": "provider", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "display_name": { + "name": "display_name", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "enabled": { + "name": "enabled", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": true + }, + "capabilities": { + "name": "capabilities", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false + }, + "settings": { + "name": "settings", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'{}'" + }, + "credentials": { + "name": "credentials", + "type": "text", + "primaryKey": false, + "notNull": false, + "autoincrement": false, + "default": "'{}'" + }, + "circuit_state": { + "name": "circuit_state", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'CLOSED'" + }, + "last_failure": { + "name": "last_failure", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "failure_count": { + "name": "failure_count", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 0 + }, + "health_status": { + "name": "health_status", + "type": "text", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "'healthy'" + }, + "last_health_check": { + "name": "last_health_check", + "type": "integer", + "primaryKey": false, + "notNull": false, + "autoincrement": false + }, + "priority": { + "name": "priority", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": 100 + }, + "created_at": { + "name": "created_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + }, + "updated_at": { + "name": "updated_at", + "type": "integer", + "primaryKey": false, + "notNull": true, + "autoincrement": false, + "default": "(unixepoch())" + } + }, + "indexes": { + "provider_configs_provider_unique": { + "name": "provider_configs_provider_unique", + "columns": [ + "provider" + ], + "isUnique": true + }, + "idx_provider_configs_enabled": { + "name": "idx_provider_configs_enabled", + "columns": [ + "enabled" + ], + "isUnique": false + }, + "idx_provider_configs_priority": { + "name": "idx_provider_configs_priority", + "columns": [ + "priority" + ], + "isUnique": false + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "checkConstraints": {} + } + }, + "views": {}, + "enums": {}, + "_meta": { + "schemas": {}, + "tables": {}, + "columns": {} + }, + "internal": { + "indexes": { + "idx_streak_user": { + "columns": { + "COALESCE(\"user_id\", -1)": { + "isExpression": true + } + } + }, + "idx_goal_user_year": { + "columns": { + "COALESCE(\"user_id\", -1)": { + "isExpression": true + } + } + } + } + } +} \ No newline at end of file diff --git a/drizzle/meta/_journal.json b/drizzle/meta/_journal.json index 78bb1175..542debb6 100644 --- a/drizzle/meta/_journal.json +++ b/drizzle/meta/_journal.json @@ -155,6 +155,13 @@ "when": 1770212841932, "tag": "0021_windy_loki", "breakpoints": true + }, + { + "idx": 22, + "version": "6", + "when": 1770333598544, + "tag": "0022_nappy_spectrum", + "breakpoints": true } ] } \ No newline at end of file diff --git a/lib/db/schema/books.ts b/lib/db/schema/books.ts index 5f9a6775..2d15b760 100644 --- a/lib/db/schema/books.ts +++ b/lib/db/schema/books.ts @@ -1,11 +1,18 @@ -import { sqliteTable, text, integer, real, index } from "drizzle-orm/sqlite-core"; +import { sqliteTable, text, integer, real, index, uniqueIndex } from "drizzle-orm/sqlite-core"; import { sql } from "drizzle-orm"; export const books = sqliteTable( "books", { id: integer("id").primaryKey({ autoIncrement: true }), - calibreId: integer("calibre_id").notNull().unique(), + // Multi-source support: calibreId is now nullable for non-Calibre books + calibreId: integer("calibre_id").unique(), + // Source indicates which provider this book came from + source: text("source", { enum: ["calibre", "manual", "hardcover", "openlibrary"] }) + .notNull() + .default("calibre"), + // Provider-specific ID (null for manual books, Calibre ID for Calibre books) + externalId: text("external_id"), title: text("title").notNull(), // Store authors as JSON array authors: text("authors", { mode: "json" }).$type().notNull().default(sql`'[]'`), @@ -14,16 +21,17 @@ export const books = sqliteTable( isbn: text("isbn"), totalPages: integer("total_pages"), addedToLibrary: integer("added_to_library", { mode: "timestamp" }).notNull().default(sql`(unixepoch())`), - lastSynced: integer("last_synced", { mode: "timestamp" }).notNull().default(sql`(unixepoch())`), + lastSynced: integer("last_synced", { mode: "timestamp" }), publisher: text("publisher"), pubDate: integer("pub_date", { mode: "timestamp" }), series: text("series"), seriesIndex: real("series_index"), // Store tags as JSON array tags: text("tags", { mode: "json" }).$type().notNull().default(sql`'[]'`), - path: text("path").notNull(), + // Path is Calibre-specific, nullable for non-Calibre books + path: text("path"), description: text("description"), - rating: integer("rating"), // 1-5 stars, synced from Calibre + rating: integer("rating"), // 1-5 stars, synced from Calibre or set manually orphaned: integer("orphaned", { mode: "boolean" }).notNull().default(false), orphanedAt: integer("orphaned_at", { mode: "timestamp" }), createdAt: integer("created_at", { mode: "timestamp" }).notNull().default(sql`(unixepoch())`), @@ -32,6 +40,12 @@ export const books = sqliteTable( (table) => ({ // Index for efficient author sorting authorSortIdx: index("idx_books_author_sort").on(table.authorSort), + // Index for source-based filtering + sourceIdx: index("idx_books_source").on(table.source), + // Unique constraint on (source, externalId) where externalId is not null + sourceExternalIdx: uniqueIndex("idx_books_source_external") + .on(table.source, table.externalId) + .where(sql`external_id IS NOT NULL`), }) ); diff --git a/lib/db/schema/index.ts b/lib/db/schema/index.ts index e20b86a7..61f2c0c3 100644 --- a/lib/db/schema/index.ts +++ b/lib/db/schema/index.ts @@ -5,3 +5,4 @@ export * from "./progress-logs"; export * from "./streaks"; export * from "./reading-goals"; export * from "./shelves"; +export * from "./provider-configs"; diff --git a/lib/db/schema/provider-configs.ts b/lib/db/schema/provider-configs.ts new file mode 100644 index 00000000..108b136e --- /dev/null +++ b/lib/db/schema/provider-configs.ts @@ -0,0 +1,70 @@ +import { sqliteTable, text, integer, index } from "drizzle-orm/sqlite-core"; +import { sql } from "drizzle-orm"; + +export const providerConfigs = sqliteTable( + "provider_configs", + { + id: integer("id").primaryKey({ autoIncrement: true }), + // Provider identifier (calibre, manual, hardcover, openlibrary) + provider: text("provider", { + enum: ["calibre", "manual", "hardcover", "openlibrary"], + }) + .notNull() + .unique(), + // Display name for UI + displayName: text("display_name").notNull(), + // Provider enabled/disabled state + enabled: integer("enabled", { mode: "boolean" }).notNull().default(true), + // Provider capabilities (JSON) + capabilities: text("capabilities", { mode: "json" }) + .$type<{ + hasSearch: boolean; + hasMetadataFetch: boolean; + hasSync: boolean; + requiresAuth: boolean; + }>() + .notNull(), + // Provider-specific settings (JSON) + settings: text("settings", { mode: "json" }) + .$type>() + .notNull() + .default(sql`'{}'`), + // Provider credentials (JSON, plaintext acceptable for local deployment) + credentials: text("credentials", { mode: "json" }) + .$type>() + .default(sql`'{}'`), + // Circuit breaker state + circuitState: text("circuit_state", { + enum: ["CLOSED", "OPEN", "HALF_OPEN"], + }) + .notNull() + .default("CLOSED"), + // Circuit breaker last failure timestamp + lastFailure: integer("last_failure", { mode: "timestamp" }), + // Circuit breaker failure count + failureCount: integer("failure_count").notNull().default(0), + // Health status (healthy/unavailable) + healthStatus: text("health_status", { enum: ["healthy", "unavailable"] }) + .notNull() + .default("healthy"), + // Last health check timestamp + lastHealthCheck: integer("last_health_check", { mode: "timestamp" }), + // Priority for provider ordering (lower = higher priority) + priority: integer("priority").notNull().default(100), + createdAt: integer("created_at", { mode: "timestamp" }) + .notNull() + .default(sql`(unixepoch())`), + updatedAt: integer("updated_at", { mode: "timestamp" }) + .notNull() + .default(sql`(unixepoch())`), + }, + (table) => ({ + // Index for enabled providers + enabledIdx: index("idx_provider_configs_enabled").on(table.enabled), + // Index for priority ordering + priorityIdx: index("idx_provider_configs_priority").on(table.priority), + }) +); + +export type ProviderConfig = typeof providerConfigs.$inferSelect; +export type NewProviderConfig = typeof providerConfigs.$inferInsert; diff --git a/lib/migrations/0022_seed_provider_configs.ts b/lib/migrations/0022_seed_provider_configs.ts new file mode 100644 index 00000000..a6d86fba --- /dev/null +++ b/lib/migrations/0022_seed_provider_configs.ts @@ -0,0 +1,147 @@ +/** + * Companion Migration: Seed Default Provider Configurations + * + * This migration seeds the provider_configs table with default configurations + * for all supported metadata providers: + * - calibre: Existing Calibre library sync + * - manual: User-entered books without external source + * - hardcover: Hardcover.app API integration + * - openlibrary: OpenLibrary.org API integration + * + * Runs only on existing databases (skipped on fresh installations where + * provider_configs will be empty). + */ + +import type { CompanionMigration } from "@/lib/db/companion-migrations"; +import { getLogger } from "@/lib/logger"; + +const logger = getLogger().child({ migration: "0022_seed_provider_configs" }); + +const migration: CompanionMigration = { + name: "0022_seed_provider_configs", + + // Only run if provider_configs table exists + requiredTables: ["provider_configs"], + + description: "Seed default provider configurations for multi-source book tracking", + + async execute(db) { + logger.info("Seeding default provider configurations..."); + + // Check if providers already exist (idempotent) + const existing = db.prepare( + "SELECT COUNT(*) as count FROM provider_configs" + ).get() as { count: number }; + + if (existing.count > 0) { + logger.info({ count: existing.count }, "Provider configs already exist, skipping seed"); + return; + } + + // Default provider configurations + const providers = [ + { + provider: "calibre", + display_name: "Calibre Library", + enabled: 1, + capabilities: JSON.stringify({ + hasSearch: false, + hasMetadataFetch: true, + hasSync: true, + requiresAuth: false, + }), + settings: JSON.stringify({}), + credentials: JSON.stringify({}), + circuit_state: "CLOSED", + failure_count: 0, + health_status: "healthy", + priority: 1, // Highest priority + }, + { + provider: "manual", + display_name: "Manual Entry", + enabled: 1, + capabilities: JSON.stringify({ + hasSearch: false, + hasMetadataFetch: false, + hasSync: false, + requiresAuth: false, + }), + settings: JSON.stringify({}), + credentials: JSON.stringify({}), + circuit_state: "CLOSED", + failure_count: 0, + health_status: "healthy", + priority: 99, // Lowest priority (fallback) + }, + { + provider: "hardcover", + display_name: "Hardcover", + enabled: 1, + capabilities: JSON.stringify({ + hasSearch: true, + hasMetadataFetch: true, + hasSync: false, + requiresAuth: false, + }), + settings: JSON.stringify({ + baseUrl: "https://hardcover.app/api/v1", + timeout: 5000, + }), + credentials: JSON.stringify({}), + circuit_state: "CLOSED", + failure_count: 0, + health_status: "healthy", + priority: 10, + }, + { + provider: "openlibrary", + display_name: "Open Library", + enabled: 1, + capabilities: JSON.stringify({ + hasSearch: true, + hasMetadataFetch: true, + hasSync: false, + requiresAuth: false, + }), + settings: JSON.stringify({ + baseUrl: "https://openlibrary.org/api", + timeout: 5000, + }), + credentials: JSON.stringify({}), + circuit_state: "CLOSED", + failure_count: 0, + health_status: "healthy", + priority: 20, + }, + ]; + + const insertStmt = db.prepare(` + INSERT INTO provider_configs ( + provider, display_name, enabled, capabilities, settings, credentials, + circuit_state, failure_count, health_status, priority + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + `); + + for (const provider of providers) { + insertStmt.run( + provider.provider, + provider.display_name, + provider.enabled, + provider.capabilities, + provider.settings, + provider.credentials, + provider.circuit_state, + provider.failure_count, + provider.health_status, + provider.priority + ); + + logger.info({ provider: provider.provider }, "Seeded provider config"); + } + + logger.info({ count: providers.length }, "Provider config seeding complete"); + } +}; + +export default migration; diff --git a/lib/providers/base/IMetadataProvider.ts b/lib/providers/base/IMetadataProvider.ts new file mode 100644 index 00000000..4c0696aa --- /dev/null +++ b/lib/providers/base/IMetadataProvider.ts @@ -0,0 +1,197 @@ +/** + * Metadata Provider Interface + * + * Defines the contract for all book metadata providers (Calibre, manual entry, + * external APIs like Hardcover and OpenLibrary). + * + * Providers declare capabilities via boolean flags and implement only the + * methods corresponding to their capabilities. + */ + +/** + * Provider health status + * + * Tracks provider availability for circuit breaker pattern: + * - healthy: Provider responding normally + * - unavailable: Provider failing (circuit open) + */ +export type ProviderHealth = "healthy" | "unavailable"; + +/** + * Book source identifier + * + * Indicates which provider a book originated from: + * - calibre: Synced from Calibre library database + * - manual: User-entered via manual entry form + * - hardcover: Fetched from Hardcover.app API + * - openlibrary: Fetched from OpenLibrary.org API + */ +export type BookSource = "calibre" | "manual" | "hardcover" | "openlibrary"; + +/** + * Provider capabilities + * + * Boolean flags indicating which operations a provider supports. + * Used for runtime validation and UI feature toggling. + */ +export interface ProviderCapabilities { + /** Can search for books by query string */ + hasSearch: boolean; + + /** Can fetch full metadata by external ID */ + hasMetadataFetch: boolean; + + /** Can sync entire library (batch import) */ + hasSync: boolean; + + /** Requires authentication credentials (API key, etc.) */ + requiresAuth: boolean; +} + +/** + * Book metadata structure + * + * Normalized metadata format returned by all providers. + * Maps to the books table schema. + */ +export interface BookMetadata { + // Core metadata + title: string; + authors: string[]; + isbn?: string; + description?: string; + + // Publication details + publisher?: string; + pubDate?: Date; + totalPages?: number; + + // Series information + series?: string; + seriesIndex?: number; + + // External identifiers + externalId?: string; // Provider-specific ID + + // Additional metadata + tags?: string[]; + coverImageUrl?: string; + rating?: number; // 1-5 stars +} + +/** + * Search result item + * + * Lightweight search result with essential metadata for selection UI. + */ +export interface SearchResult { + externalId: string; + title: string; + authors: string[]; + isbn?: string; + coverImageUrl?: string; + publisher?: string; + pubDate?: Date; +} + +/** + * Sync operation result + * + * Statistics from batch library synchronization. + */ +export interface SyncResult { + added: number; + updated: number; + removed: number; + errors: number; +} + +/** + * Metadata Provider Interface + * + * Core interface for all book metadata providers. Providers implement + * only the methods corresponding to their declared capabilities. + * + * @example + * ```typescript + * class ManualProvider implements IMetadataProvider { + * id = 'manual'; + * name = 'Manual Entry'; + * capabilities = { + * hasSearch: false, + * hasMetadataFetch: false, + * hasSync: false, + * requiresAuth: false, + * }; + * + * async healthCheck() { + * return 'healthy' as const; + * } + * } + * ``` + */ +export interface IMetadataProvider { + /** Unique provider identifier (matches BookSource) */ + readonly id: BookSource; + + /** Human-readable display name */ + readonly name: string; + + /** Provider capabilities declaration */ + readonly capabilities: ProviderCapabilities; + + /** + * Search for books by query + * + * Only implement if capabilities.hasSearch = true + * + * @param query - Search query string (title, author, ISBN, etc.) + * @returns Array of search results + * @throws Error if provider unavailable or API error + */ + search?(query: string): Promise; + + /** + * Fetch full metadata by external ID + * + * Only implement if capabilities.hasMetadataFetch = true + * + * @param externalId - Provider-specific book identifier + * @returns Complete book metadata + * @throws Error if book not found or provider unavailable + */ + fetchMetadata?(externalId: string): Promise; + + /** + * Sync entire library (batch import) + * + * Only implement if capabilities.hasSync = true + * + * @returns Sync statistics + * @throws Error if sync fails + */ + sync?(): Promise; + + /** + * Check provider health + * + * Must be implemented by all providers. Used for circuit breaker pattern + * to detect and isolate failing providers. + * + * @returns Current health status + */ + healthCheck(): Promise; +} + +/** + * Provider registry entry + * + * Internal type for provider registry with runtime state. + */ +export interface ProviderRegistryEntry { + provider: IMetadataProvider; + enabled: boolean; + priority: number; + lastHealthCheck?: Date; + healthStatus: ProviderHealth; +} diff --git a/lib/providers/base/ProviderRegistry.ts b/lib/providers/base/ProviderRegistry.ts new file mode 100644 index 00000000..e8a8667b --- /dev/null +++ b/lib/providers/base/ProviderRegistry.ts @@ -0,0 +1,282 @@ +/** + * Provider Registry + * + * Central registry for all metadata providers. Handles provider discovery, + * validation, and lookup operations. + * + * See: specs/003-non-calibre-books/research.md (Decision 1: Provider Interface) + */ + +import { getLogger } from "@/lib/logger"; +import type { + IMetadataProvider, + ProviderRegistryEntry, + BookSource, + ProviderHealth, +} from "./IMetadataProvider"; + +const logger = getLogger().child({ module: "provider-registry" }); + +/** + * Provider Registry + * + * Singleton registry for managing metadata providers. Validates provider + * implementations and provides lookup/discovery operations. + * + * @example + * ```typescript + * // Register provider + * ProviderRegistry.register(calibreProvider); + * + * // Get provider + * const provider = ProviderRegistry.get('calibre'); + * + * // Get enabled providers sorted by priority + * const providers = ProviderRegistry.getEnabled(); + * ``` + */ +export class ProviderRegistry { + private static providers = new Map(); + private static initialized = false; + + /** + * Register a metadata provider + * + * Validates provider implementation against declared capabilities. + * + * @param provider - Provider instance to register + * @param options - Registration options + * @throws Error if provider invalid or already registered + */ + static register( + provider: IMetadataProvider, + options: { + enabled?: boolean; + priority?: number; + } = {} + ): void { + const { enabled = true, priority = 100 } = options; + + // Validate provider + this.validateProvider(provider); + + // Check for duplicates + if (this.providers.has(provider.id)) { + throw new Error(`Provider '${provider.id}' already registered`); + } + + // Register provider + this.providers.set(provider.id, { + provider, + enabled, + priority, + healthStatus: "healthy", + }); + + logger.info( + { + providerId: provider.id, + capabilities: provider.capabilities, + enabled, + priority, + }, + "Registered provider" + ); + } + + /** + * Validate provider implementation + * + * Ensures provider implements methods corresponding to declared capabilities. + * + * @internal + */ + private static validateProvider(provider: IMetadataProvider): void { + // Check required fields + if (!provider.id || !provider.name || !provider.capabilities) { + throw new Error( + `Provider missing required fields: id, name, capabilities` + ); + } + + // Validate healthCheck exists + if (typeof provider.healthCheck !== "function") { + throw new Error( + `Provider '${provider.id}' must implement healthCheck method` + ); + } + + // Validate capability methods + const { capabilities } = provider; + + if (capabilities.hasSearch && typeof provider.search !== "function") { + throw new Error( + `Provider '${provider.id}' declares hasSearch but search method not implemented` + ); + } + + if ( + capabilities.hasMetadataFetch && + typeof provider.fetchMetadata !== "function" + ) { + throw new Error( + `Provider '${provider.id}' declares hasMetadataFetch but fetchMetadata method not implemented` + ); + } + + if (capabilities.hasSync && typeof provider.sync !== "function") { + throw new Error( + `Provider '${provider.id}' declares hasSync but sync method not implemented` + ); + } + } + + /** + * Get provider by ID + * + * @param id - Provider identifier + * @returns Provider instance or undefined + */ + static get(id: BookSource): IMetadataProvider | undefined { + return this.providers.get(id)?.provider; + } + + /** + * Get provider entry (includes runtime state) + * + * @internal + */ + static getEntry(id: BookSource): ProviderRegistryEntry | undefined { + return this.providers.get(id); + } + + /** + * Get all registered providers + * + * @returns Array of all providers (unsorted) + */ + static getAll(): IMetadataProvider[] { + return Array.from(this.providers.values()).map((entry) => entry.provider); + } + + /** + * Get enabled providers sorted by priority + * + * @returns Array of enabled providers (ascending priority) + */ + static getEnabled(): IMetadataProvider[] { + return Array.from(this.providers.values()) + .filter((entry) => entry.enabled) + .sort((a, b) => a.priority - b.priority) + .map((entry) => entry.provider); + } + + /** + * Get providers with specific capability + * + * @param capability - Capability name to filter by + * @returns Array of providers with capability (sorted by priority) + */ + static getByCapability( + capability: keyof IMetadataProvider["capabilities"] + ): IMetadataProvider[] { + return this.getEnabled().filter( + (provider) => provider.capabilities[capability] + ); + } + + /** + * Update provider health status + * + * @param id - Provider identifier + * @param status - New health status + */ + static updateHealth(id: BookSource, status: ProviderHealth): void { + const entry = this.providers.get(id); + if (entry) { + entry.healthStatus = status; + entry.lastHealthCheck = new Date(); + logger.debug({ providerId: id, status }, "Updated provider health"); + } + } + + /** + * Enable/disable provider at runtime + * + * @param id - Provider identifier + * @param enabled - Enable state + */ + static setEnabled(id: BookSource, enabled: boolean): void { + const entry = this.providers.get(id); + if (entry) { + entry.enabled = enabled; + logger.info({ providerId: id, enabled }, "Updated provider enabled state"); + } + } + + /** + * Update provider priority + * + * @param id - Provider identifier + * @param priority - New priority value (lower = higher priority) + */ + static setPriority(id: BookSource, priority: number): void { + const entry = this.providers.get(id); + if (entry) { + entry.priority = priority; + logger.info({ providerId: id, priority }, "Updated provider priority"); + } + } + + /** + * Check if provider exists + * + * @param id - Provider identifier + * @returns True if provider registered + */ + static has(id: BookSource): boolean { + return this.providers.has(id); + } + + /** + * Unregister provider (primarily for testing) + * + * @param id - Provider identifier + * @internal + */ + static unregister(id: BookSource): void { + this.providers.delete(id); + logger.debug({ providerId: id }, "Unregistered provider"); + } + + /** + * Clear all providers (primarily for testing) + * + * @internal + */ + static clear(): void { + this.providers.clear(); + this.initialized = false; + logger.debug("Cleared provider registry"); + } + + /** + * Mark registry as initialized + * + * Used to prevent re-initialization in production. + * + * @internal + */ + static markInitialized(): void { + this.initialized = true; + } + + /** + * Check if registry is initialized + * + * @returns True if initialized + */ + static isInitialized(): boolean { + return this.initialized; + } +} diff --git a/lib/repositories/book.repository.ts b/lib/repositories/book.repository.ts index b7187aff..a76d9d0a 100644 --- a/lib/repositories/book.repository.ts +++ b/lib/repositories/book.repository.ts @@ -109,7 +109,9 @@ export class BookRepository extends BaseRepository // Build map: calibreId -> Book const booksMap = new Map(); for (const book of results) { - booksMap.set(book.calibreId, book); + if (book.calibreId !== null) { + booksMap.set(book.calibreId, book); + } } return booksMap; @@ -140,6 +142,8 @@ export class BookRepository extends BaseRepository // Book fields bookId: books.id, calibreId: books.calibreId, + source: books.source, + externalId: books.externalId, title: books.title, authors: books.authors, authorSort: books.authorSort, @@ -259,6 +263,8 @@ export class BookRepository extends BaseRepository const book: Book = { id: result.bookId, calibreId: result.calibreId, + source: result.source, + externalId: result.externalId, title: result.title, authors: result.authors, authorSort: result.authorSort, diff --git a/lib/services/session.service.ts b/lib/services/session.service.ts index cf0d8a6a..e57db5ae 100644 --- a/lib/services/session.service.ts +++ b/lib/services/session.service.ts @@ -561,9 +561,11 @@ export class SessionService { } try { - // Sync to Calibre first (best effort) - calibreService.updateRating(book.calibreId, rating); - logger.info({ bookId, calibreId: book.calibreId, rating }, "Synced rating to Calibre"); + // Sync to Calibre first (best effort) - only for Calibre books + if (book.source === 'calibre' && book.calibreId !== null) { + calibreService.updateRating(book.calibreId, rating); + logger.info({ bookId, calibreId: book.calibreId, rating }, "Synced rating to Calibre"); + } } catch (calibreError) { // Log error but continue with Tome database update logger.error({ err: calibreError, bookId }, "Failed to sync rating to Calibre"); From 1bfb3e19862a07eb7a493cb6da92139d7e74aeee Mon Sep 17 00:00:00 2001 From: Mason Fox Date: Thu, 5 Feb 2026 18:28:34 -0500 Subject: [PATCH 05/89] feat(spec-003): Phase 2 Part 1 - Repository layer extensions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Created ProviderConfigRepository with full CRUD operations (T009) - Extended BookRepository with source filtering methods (T010): * findBySource() - get all books from a specific source * findBySourceAndExternalId() - check for existing external books * countBySource() - count books per source - Added source filter parameter to findWithFilters() (T011) - Updated findNotInCalibreIds() to only check Calibre books (T012) * CRITICAL: Prevents orphaning manual/external books during sync Repository Layer Complete: ✅ T009-T012: Source-aware repository methods ✅ Multi-source book tracking foundation ready Next: T013-T020 - Service layer and provider implementations --- lib/repositories/book.repository.ts | 70 ++- .../provider-config.repository.ts | 152 +++++++ specs/003-non-calibre-books/tasks.md | 400 ++++++++++++++++++ 3 files changed, 621 insertions(+), 1 deletion(-) create mode 100644 lib/repositories/provider-config.repository.ts create mode 100644 specs/003-non-calibre-books/tasks.md diff --git a/lib/repositories/book.repository.ts b/lib/repositories/book.repository.ts index a76d9d0a..1fadb94d 100644 --- a/lib/repositories/book.repository.ts +++ b/lib/repositories/book.repository.ts @@ -18,6 +18,7 @@ export interface BookFilter { shelfIds?: number[]; // Filter by books on specific shelves (OR logic - book must be on ANY shelf) excludeShelfId?: number; // Exclude books on this shelf (used for "Add Books to Shelf" modal) noTags?: boolean; // Filter for books without any tags + source?: Book["source"] | Book["source"][]; // Filter by source(s) } export interface BookWithStatus extends Book { @@ -70,6 +71,57 @@ export class BookRepository extends BaseRepository .get(); } + /** + * Find all books from a specific source + * + * @param source - Book source (calibre, manual, hardcover, openlibrary) + * @returns Array of books from the specified source + */ + async findBySource(source: Book["source"]): Promise { + return this.getDatabase() + .select() + .from(books) + .where(eq(books.source, source)) + .all(); + } + + /** + * Find a book by source and external ID + * + * Used to check if a book from an external provider already exists + * before creating a duplicate. + * + * @param source - Book source + * @param externalId - Provider-specific book identifier + * @returns Book if found, undefined otherwise + */ + async findBySourceAndExternalId( + source: Book["source"], + externalId: string + ): Promise { + return this.getDatabase() + .select() + .from(books) + .where(and(eq(books.source, source), eq(books.externalId, externalId))) + .get(); + } + + /** + * Count books by source + * + * @param source - Book source to count + * @returns Number of books from the specified source + */ + async countBySource(source: Book["source"]): Promise { + const result = this.getDatabase() + .select({ count: sql`COUNT(*)` }) + .from(books) + .where(eq(books.source, source)) + .get(); + + return result?.count ?? 0; + } + /** * Find multiple books by IDs */ @@ -339,6 +391,19 @@ export class BookRepository extends BaseRepository conditions.push(isNotOrphaned()); } + // Source filter (single source or array of sources) + if (filters.source) { + if (Array.isArray(filters.source)) { + // Multiple sources - book must be from ANY of the sources (OR logic) + if (filters.source.length > 0) { + conditions.push(inArray(books.source, filters.source)); + } + } else { + // Single source + conditions.push(eq(books.source, filters.source)); + } + } + // Search filter if (filters.search) { const searchPattern = `%${filters.search}%`; @@ -611,7 +676,9 @@ export class BookRepository extends BaseRepository } /** - * Find books not in a list of calibreIds (for orphaning) + * Find books that are NOT in the given Calibre IDs (for orphan detection during sync) + * + * CRITICAL: Only returns Calibre-sourced books to prevent orphaning manual/external books */ async findNotInCalibreIds(calibreIds: number[]): Promise { // SAFETY: If calibreIds is empty, return empty array to prevent mass orphaning @@ -625,6 +692,7 @@ export class BookRepository extends BaseRepository .from(books) .where( and( + eq(books.source, "calibre"), // CRITICAL: Only check Calibre books sql`${books.calibreId} NOT IN ${sql.raw(`(${calibreIds.join(",")})`)}`, isNotOrphaned() ) diff --git a/lib/repositories/provider-config.repository.ts b/lib/repositories/provider-config.repository.ts new file mode 100644 index 00000000..2e4c2060 --- /dev/null +++ b/lib/repositories/provider-config.repository.ts @@ -0,0 +1,152 @@ +/** + * Provider Config Repository + * + * Manages CRUD operations for metadata provider configurations. + * Stores runtime settings, credentials, health status, and circuit breaker state. + */ + +import { eq } from "drizzle-orm"; +import { BaseRepository } from "./base.repository"; +import { + providerConfigs, + ProviderConfig, + NewProviderConfig, +} from "@/lib/db/schema/provider-configs"; +import type { BookSource } from "@/lib/providers/base/IMetadataProvider"; + +export class ProviderConfigRepository extends BaseRepository< + ProviderConfig, + NewProviderConfig, + typeof providerConfigs +> { + constructor() { + super(providerConfigs); + } + + protected getTable() { + return providerConfigs; + } + + /** + * Find provider config by provider ID + */ + async findByProvider(provider: BookSource): Promise { + return this.getDatabase() + .select() + .from(providerConfigs) + .where(eq(providerConfigs.provider, provider)) + .get(); + } + + /** + * Find all enabled providers ordered by priority + */ + async findEnabled(): Promise { + return this.getDatabase() + .select() + .from(providerConfigs) + .where(eq(providerConfigs.enabled, true)) + .orderBy(providerConfigs.priority) + .all(); + } + + /** + * Update provider enabled state + */ + async setEnabled(provider: BookSource, enabled: boolean): Promise { + const existing = await this.findByProvider(provider); + if (!existing) { + return undefined; + } + + return this.update(existing.id, { enabled }); + } + + /** + * Update provider health status + */ + async updateHealth( + provider: BookSource, + healthStatus: ProviderConfig["healthStatus"], + lastHealthCheck: Date + ): Promise { + const existing = await this.findByProvider(provider); + if (!existing) { + return undefined; + } + + return this.update(existing.id, { + healthStatus, + lastHealthCheck, + }); + } + + /** + * Update circuit breaker state + */ + async updateCircuitState( + provider: BookSource, + circuitState: ProviderConfig["circuitState"], + failureCount?: number, + lastFailure?: Date + ): Promise { + const existing = await this.findByProvider(provider); + if (!existing) { + return undefined; + } + + return this.update(existing.id, { + circuitState, + failureCount: failureCount ?? existing.failureCount, + lastFailure: lastFailure ?? existing.lastFailure, + }); + } + + /** + * Reset circuit breaker failure count + */ + async resetFailureCount(provider: BookSource): Promise { + const existing = await this.findByProvider(provider); + if (!existing) { + return undefined; + } + + return this.update(existing.id, { + failureCount: 0, + lastFailure: null, + } as Partial); + } + + /** + * Update provider settings + */ + async updateSettings( + provider: BookSource, + settings: Record + ): Promise { + const existing = await this.findByProvider(provider); + if (!existing) { + return undefined; + } + + return this.update(existing.id, { settings }); + } + + /** + * Update provider credentials + */ + async updateCredentials( + provider: BookSource, + credentials: Record + ): Promise { + const existing = await this.findByProvider(provider); + if (!existing) { + return undefined; + } + + return this.update(existing.id, { credentials }); + } +} + +// Export singleton instance +export const providerConfigRepository = new ProviderConfigRepository(); diff --git a/specs/003-non-calibre-books/tasks.md b/specs/003-non-calibre-books/tasks.md new file mode 100644 index 00000000..5ff6fa63 --- /dev/null +++ b/specs/003-non-calibre-books/tasks.md @@ -0,0 +1,400 @@ +# Implementation Tasks: Support Non-Calibre Books + +**Branch**: `003-non-calibre-books` | **Date**: 2026-02-05 +**Spec**: [spec.md](./spec.md) | **Plan**: [plan.md](./plan.md) + +## Overview + +This document breaks down the multi-source book tracking feature into executable tasks organized by user story. Each phase represents an independently testable increment of functionality. + +**Total Tasks**: 89 +**Implementation Approach**: Incremental delivery by priority (P1 → P2 → P3) +**MVP Scope**: User Story 1 + User Story 2 (P1 stories - manual books with sync isolation) + +--- + +## Task Summary by Phase + +| Phase | User Story | Priority | Task Count | Parallelizable | +|-------|------------|----------|------------|----------------| +| 1 | Setup | - | 8 | 3 | +| 2 | Foundational | - | 12 | 6 | +| 3 | Manual Book Addition | P1 | 18 | 10 | +| 4 | Library Sync Isolation | P1 | 9 | 4 | +| 5 | Source-Based Filtering | P2 | 8 | 5 | +| 6 | Source Migration & Duplicates | P2 | 12 | 6 | +| 7 | Federated Search | P3 | 16 | 9 | +| 8 | Polish & Cross-Cutting | - | 6 | 3 | + +--- + +## Phase 1: Setup (Infrastructure) + +**Goal**: Establish database schema, migrations, and core provider infrastructure + +**Prerequisites**: None (starting point) + +### Database Schema & Migrations + +- [X] T001 Generate Drizzle schema migration for books table (add source, externalId, make calibreId nullable) in drizzle/0022_nappy_spectrum.sql +- [X] T002 Create provider_configs schema in lib/db/schema/provider-configs.ts +- [X] T003 Create companion migration to populate source='calibre' for existing books in lib/migrations/0022_seed_provider_configs.ts +- [X] T004 Update books schema exports in lib/db/schema/books.ts (add source, externalId fields with enums and indexes) +- [X] T005 Seed provider_configs table with default Hardcover and OpenLibrary configs in migration + +### Provider Architecture Foundation + +- [X] T006 [P] Create IMetadataProvider interface with capability flags in lib/providers/base/IMetadataProvider.ts +- [X] T007 [P] Implement ProviderRegistry for provider discovery and registration in lib/providers/base/ProviderRegistry.ts +- [X] T008 [P] Create provider types and shared utilities in lib/providers/base/IMetadataProvider.ts (types integrated with interface) + +--- + +## Phase 2: Foundational (Blocking Prerequisites) + +**Goal**: Build core services and repositories that all user stories depend on + +**Prerequisites**: Phase 1 complete + +### Repository Layer + +- [X] T009 [P] Create ProviderConfigRepository with CRUD operations in lib/repositories/provider-config.repository.ts +- [X] T010 [P] Extend BookRepository with source filtering methods (findBySource, findBySourceAndExternalId, countBySource) in lib/repositories/book.repository.ts +- [X] T011 [P] Add source parameter to BookRepository.findWithFilters() in lib/repositories/book.repository.ts +- [X] T012 Update BookRepository.findNotInCalibreIds() to filter by source='calibre' in lib/repositories/book.repository.ts + +### Service Layer Foundation + +- [ ] T013 [P] Create CircuitBreakerService with state machine (CLOSED/OPEN/HALF_OPEN) in lib/services/circuit-breaker.service.ts +- [ ] T014 [P] Implement ProviderService for provider orchestration in lib/services/provider.service.ts +- [ ] T015 [P] Create MigrationService for source migration workflows in lib/services/migration.service.ts + +### Provider Implementations (Stubs) + +- [ ] T016 [P] Create ManualProvider stub (always enabled, no external API) in lib/providers/manual.provider.ts +- [ ] T017 [P] Refactor existing Calibre sync to CalibreProvider implementation in lib/providers/calibre.provider.ts +- [ ] T018 [P] Create HardcoverProvider stub with auth handling in lib/providers/hardcover.provider.ts +- [ ] T019 [P] Create OpenLibraryProvider stub (public API, no auth) in lib/providers/openlibrary.provider.ts +- [ ] T020 Register all providers in ProviderRegistry singleton in lib/providers/base/ProviderRegistry.ts + +--- + +## Phase 3: User Story 1 - Manual Book Addition (P1) + +**Goal**: Enable users to manually add books with validation and duplicate warnings + +**Independent Test**: Add manual book via UI → log progress → view in library with source badge + +**Prerequisites**: Phase 2 complete + +### Backend - Manual Book Creation + +- [ ] T021 [P] [US1] Extend POST /api/books to accept manual book creation (source='manual', no calibreId) in app/api/books/route.ts +- [ ] T022 [P] [US1] Add validation for manual books (title, author, pageCount required, pageCount 1-10000) in lib/services/book.service.ts +- [ ] T023 [US1] Implement duplicate detection service using Levenshtein distance (>85% threshold) in lib/services/duplicate-detection.service.ts +- [ ] T024 [US1] Integrate duplicate check into manual book creation flow (warn but allow proceed) in lib/services/book.service.ts + +### Backend - Validation & Error Handling + +- [ ] T025 [P] [US1] Create validation schemas for manual book input using Zod in lib/validation/manual-book.schema.ts +- [ ] T026 [P] [US1] Add real-time validation endpoint POST /api/books/validate in app/api/books/validate/route.ts +- [ ] T027 [US1] Implement server-side validation for optional fields (ISBN, publisher, publicationDate, description, coverImageUrl) in lib/validation/manual-book.schema.ts + +### Frontend - Manual Book Form + +- [ ] T028 [P] [US1] Create ManualBookForm component with required fields in components/books/ManualBookForm.tsx +- [ ] T029 [P] [US1] Add real-time validation UI with error messages in components/books/ManualBookForm.tsx +- [ ] T030 [P] [US1] Create DuplicateWarning modal component in components/providers/DuplicateWarning.tsx +- [ ] T031 [US1] Integrate duplicate detection API call on form submission in components/books/ManualBookForm.tsx +- [ ] T032 [US1] Add "Add Manual Book" button to library page header in app/library/page.tsx + +### Frontend - Source Display + +- [ ] T033 [P] [US1] Create ProviderBadge component for source indicators in components/providers/ProviderBadge.tsx +- [ ] T034 [P] [US1] Add source badge to book cards in library view in components/books/BookCard.tsx +- [ ] T035 [P] [US1] Add source badge to book detail page in app/books/[id]/page.tsx + +### Integration & Verification + +- [ ] T036 [US1] Update GET /api/books to include source field in response in app/api/books/route.ts +- [ ] T037 [US1] Verify manual books support all existing features (progress, sessions, streaks) via existing API routes +- [ ] T038 [US1] Test manual book creation end-to-end (UI → validation → save → display) + +--- + +## Phase 4: User Story 2 - Library Sync Isolation (P1) + +**Goal**: Ensure Calibre sync operations only affect Calibre-sourced books + +**Independent Test**: Add manual books → run Calibre sync that removes books → verify manual books untouched + +**Prerequisites**: Phase 3 complete (manual books exist to test isolation) + +### Calibre Sync Isolation + +- [ ] T039 [US2] Update syncCalibreLibrary() to filter by source='calibre' in lib/sync-service.ts +- [ ] T040 [US2] Update orphaned book detection to only mark Calibre books in lib/sync-service.ts +- [ ] T041 [US2] Add source='calibre' filter to book creation/update during sync in lib/sync-service.ts +- [ ] T042 [US2] Update CalibreProvider to respect source boundaries in lib/providers/calibre.provider.ts + +### Testing & Validation + +- [ ] T043 [P] [US2] Create integration test: manual book + Calibre sync → verify isolation in __tests__/integration/sync-isolation.test.ts +- [ ] T044 [P] [US2] Create test case: Calibre removes book → only Calibre books orphaned in __tests__/integration/sync-isolation.test.ts +- [ ] T045 [P] [US2] Create test case: Calibre adds book → manual books unchanged in __tests__/integration/sync-isolation.test.ts +- [ ] T046 [P] [US2] Create test case: same title in Calibre + manual → both exist independently in __tests__/integration/sync-isolation.test.ts + +### Logging & Observability + +- [ ] T047 [US2] Add Pino logging for sync operations with source filtering details in lib/sync-service.ts + +--- + +## Phase 5: User Story 3 - Source-Based Filtering (P2) + +**Goal**: Allow users to filter library by book source(s) + +**Independent Test**: Add books from multiple sources → apply filters → verify correct subset displayed + +**Prerequisites**: Phase 4 complete (multiple source types exist) + +### Backend - Filtering API + +- [ ] T048 [P] [US3] Extend GET /api/books to accept source[] query parameter in app/api/books/route.ts +- [ ] T049 [P] [US3] Update BookRepository.findWithFilters() to handle multi-source filtering in lib/repositories/book.repository.ts +- [ ] T050 [US3] Add source counts to stats API GET /api/stats/overview in app/api/stats/overview/route.ts + +### Frontend - Filter UI + +- [ ] T051 [P] [US3] Add source filter to BookFilters component (multi-select dropdown) in components/books/BookFilters.tsx +- [ ] T052 [P] [US3] Update useLibraryData hook to handle source filters in hooks/useLibraryData.ts +- [ ] T053 [P] [US3] Persist source filter state in URL params in app/library/page.tsx +- [ ] T054 [US3] Add "Clear Filters" button to BookFilters component in components/books/BookFilters.tsx + +### Performance & Optimization + +- [ ] T055 [US3] Verify source filtering performance with 10k book test dataset (target <3s) + +--- + +## Phase 6: User Story 5 - Source Migration & Duplicate Handling (P2) + +**Goal**: Enable upgrading manual books to external provider books with duplicate detection + +**Independent Test**: Add manual book → search Hardcover → select match → verify upgrade with data preservation + +**Prerequisites**: Phase 5 complete, Phase 7 (Hardcover provider) partially complete + +### Backend - Migration Service + +- [ ] T056 [P] [US5] Implement migrateSource() with transactional updates in lib/services/migration.service.ts +- [ ] T057 [P] [US5] Add pessimistic locking (FOR UPDATE) to migration operations in lib/services/migration.service.ts +- [ ] T058 [US5] Implement migration validation rules (only manual→external, no cross-provider) in lib/services/migration.service.ts +- [ ] T059 [US5] Add logging for migration events (FR-021b) in lib/services/migration.service.ts + +### Backend - Duplicate Detection for Providers + +- [ ] T060 [P] [US5] Extend duplicate detection to scope by target provider (FR-016e) in lib/services/duplicate-detection.service.ts +- [ ] T061 [P] [US5] Create POST /api/migration/[bookId] endpoint for source migration in app/api/migration/[bookId]/route.ts +- [ ] T062 [US5] Add metadata diff comparison for user confirmation in lib/services/migration.service.ts + +### Frontend - Migration UI + +- [ ] T063 [P] [US5] Create SourceMigrationDialog component with [Upgrade] [Create Duplicate] options in components/providers/SourceMigrationDialog.tsx +- [ ] T064 [P] [US5] Add metadata comparison view (old vs. new values) in components/providers/MetadataComparisonView.tsx +- [ ] T065 [US5] Integrate migration dialog into book detail page for manual books in app/books/[id]/page.tsx +- [ ] T066 [US5] Show migration history/log in book detail page in app/books/[id]/page.tsx + +### Testing & Verification + +- [ ] T067 [US5] Create integration test: manual→hardcover migration preserves all data in __tests__/integration/source-migration.test.ts + +--- + +## Phase 7: User Story 4 - Federated Metadata Search (P3) + +**Goal**: Search multiple external providers simultaneously with graceful degradation + +**Independent Test**: Search for book → verify results from multiple providers → select result → book created + +**Prerequisites**: Phase 2 complete (provider infrastructure ready) + +### Backend - Search Service + +- [ ] T068 [P] [US4] Implement SearchService.federatedSearch() with Promise.allSettled in lib/services/search.service.ts +- [ ] T069 [P] [US4] Add per-provider 5-second timeout using AbortSignal in lib/services/search.service.ts +- [ ] T070 [P] [US4] Implement search result caching (5min TTL, invalidate on config change) in lib/services/search.service.ts +- [ ] T071 [US4] Add result sorting by hardcoded priority (Hardcover → OpenLibrary) in lib/services/search.service.ts +- [ ] T072 [US4] Implement graceful fallback to manual entry on all-provider failure in lib/services/search.service.ts + +### Backend - Provider Search Implementation + +- [ ] T073 [P] [US4] Implement HardcoverProvider.search() with retry logic in lib/providers/hardcover.provider.ts +- [ ] T074 [P] [US4] Implement OpenLibraryProvider.search() with error handling in lib/providers/openlibrary.provider.ts +- [ ] T075 [P] [US4] Add rate limit detection and circuit breaker integration in lib/providers/hardcover.provider.ts +- [ ] T076 [US4] Create POST /api/providers/search endpoint for federated search in app/api/providers/search/route.ts + +### Backend - Provider Configuration + +- [ ] T077 [P] [US4] Create GET /api/providers endpoint to list all providers with status in app/api/providers/route.ts +- [ ] T078 [P] [US4] Create PATCH /api/providers/[providerId]/config for runtime configuration in app/api/providers/[providerId]/config/route.ts +- [ ] T079 [US4] Implement provider enable/disable without restart (NFR-005) in lib/services/provider.service.ts + +### Frontend - Search UI + +- [ ] T080 [P] [US4] Create FederatedSearchModal component with provider results in components/providers/FederatedSearchModal.tsx +- [ ] T081 [P] [US4] Add provider badges to search results in components/providers/FederatedSearchModal.tsx +- [ ] T082 [P] [US4] Implement editable metadata form after result selection in components/providers/FederatedSearchModal.tsx +- [ ] T083 [US4] Add fallback UI when all providers fail/timeout in components/providers/FederatedSearchModal.tsx + +### Testing & Performance + +- [ ] T084 [US4] Create integration test: federated search with 2 providers < 6 seconds in __tests__/integration/federated-search.test.ts + +--- + +## Phase 8: Polish & Cross-Cutting Concerns + +**Goal**: Settings UI, provider health monitoring, documentation + +**Prerequisites**: All user stories complete + +### Provider Management UI + +- [ ] T085 [P] Create provider settings page in app/settings/providers/page.tsx +- [ ] T086 [P] Add provider enable/disable toggles in components/settings/ProviderToggles.tsx +- [ ] T087 [P] Create API key configuration form for Hardcover in components/settings/ProviderCredentials.tsx + +### Documentation + +- [ ] T088 Update ARCHITECTURE.md with multi-source support details in docs/ARCHITECTURE.md +- [ ] T089 Create ADRs for provider architecture, circuit breakers, migration strategy in docs/ADRs/ + +--- + +## Dependencies & Execution Order + +### Critical Path (Must Complete in Order) + +``` +Phase 1 (Setup) + ↓ +Phase 2 (Foundational) + ↓ +Phase 3 (US1: Manual Books) + ↓ +Phase 4 (US2: Sync Isolation) + ↓ +Phase 5 (US3: Source Filtering) ← Can start after Phase 3 + ↓ +Phase 6 (US5: Migration) ← Requires Phase 7 providers + ↓ +Phase 7 (US4: Federated Search) ← Can start after Phase 2 + ↓ +Phase 8 (Polish) +``` + +### User Story Dependencies + +| Story | Depends On | Reason | +|-------|------------|--------| +| US1 (Manual Books) | Phase 2 | Needs repositories and services | +| US2 (Sync Isolation) | US1 | Needs manual books to test isolation | +| US3 (Source Filtering) | US1 | Needs multiple sources to filter | +| US4 (Federated Search) | Phase 2 | Independent - only needs provider infrastructure | +| US5 (Migration) | US1, US4 | Needs manual books + external providers | + +### Parallel Execution Opportunities + +**Within Phase 2 (Foundational)**: +- Tasks T009-T012 (repositories) can run in parallel +- Tasks T013-T015 (services) can run in parallel +- Tasks T016-T020 (provider stubs) can run in parallel + +**Within Phase 3 (US1)**: +- Tasks T021-T022 (backend creation) parallel +- Tasks T025-T027 (validation) parallel to T021-T022 +- Tasks T028-T032 (frontend forms) parallel after T021 complete +- Tasks T033-T035 (frontend badges) parallel to T028-T032 + +**Within Phase 7 (US4)**: +- Tasks T068-T072 (search service) parallel +- Tasks T073-T076 (provider implementations) parallel to T068-T072 +- Tasks T077-T079 (provider config) parallel to T073-T076 +- Tasks T080-T083 (frontend) parallel after T076 complete + +--- + +## MVP Definition (Minimum Viable Product) + +**Scope**: Phase 1 + Phase 2 + Phase 3 + Phase 4 + +**Delivers**: +- ✅ Manual book addition with validation +- ✅ Duplicate warnings (but allow proceed) +- ✅ Source badges in UI +- ✅ Calibre sync isolation (data integrity) +- ✅ All existing Tome features work for manual books + +**Out of MVP**: +- ❌ Source filtering (P2) +- ❌ Source migration (P2) +- ❌ Federated search (P3) +- ❌ External providers (Hardcover, OpenLibrary) + +**MVP Testing**: After Phase 4, user can: +1. Add manual book via UI +2. Log progress for manual book +3. Run Calibre sync +4. Verify manual book untouched by sync +5. View manual books alongside Calibre books with clear source indicators + +--- + +## Implementation Strategy + +### Phase-Based Delivery + +1. **Complete Phase 1** (Setup): Database ready for multi-source +2. **Complete Phase 2** (Foundational): Services and repositories ready +3. **Ship MVP** (Phase 3 + Phase 4): Manual books + sync isolation +4. **Iterate** (Phase 5-8): Add advanced features incrementally + +### Testing Strategy + +- **Unit tests**: Repositories, services, providers (use `setDatabase()` pattern) +- **Integration tests**: Multi-source scenarios, sync isolation, migration +- **Manual testing**: UI flows for each user story +- **Performance tests**: Source filtering (<3s), federated search (<6s) + +### Risk Mitigation + +- **Migration risk**: Companion migration validates data before schema change +- **Sync isolation**: Extensive testing with mixed-source libraries +- **Provider failures**: Circuit breaker prevents cascading failures +- **Data loss**: Source migration uses transactions + pessimistic locking + +--- + +## Validation Checklist + +Before marking feature complete: + +- [ ] All 89 tasks completed and checked off +- [ ] Each user story independently testable +- [ ] All existing tests still pass (2000+ tests) +- [ ] Performance benchmarks met: + - [ ] Federated search < 6 seconds + - [ ] Source filtering < 3 seconds (10k books) + - [ ] Source migration < 2 seconds + - [ ] Circuit breaker overhead < 5ms +- [ ] Constitution compliance verified: + - [ ] Zero external dependencies + - [ ] Repository pattern followed + - [ ] Calibre read-only (except ratings) + - [ ] History preserved (no deletions) +- [ ] Documentation updated (ARCHITECTURE.md, ADRs) + +--- + +**Format Validation**: ✅ All tasks follow checklist format (checkbox, ID, optional labels, file paths) From 53ecac2cb3ac06b66aa0e0790eaaade579e9761d Mon Sep 17 00:00:00 2001 From: Mason Fox Date: Thu, 5 Feb 2026 18:30:26 -0500 Subject: [PATCH 06/89] feat(spec-003): Phase 2 Part 2 - Circuit breaker and manual provider MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Implemented CircuitBreakerService with state machine (T013) - Created ManualProvider stub (T016) - Updated tasks.md to reflect progress Circuit Breaker Features: - State machine: CLOSED→OPEN→HALF_OPEN transitions - Failure tracking with configurable thresholds - 60s cooldown period before retry - Persistence via providerConfigRepository Next: T014-T020 (ProviderService, remaining provider stubs) --- lib/providers/manual.provider.ts | 37 ++++ lib/services/circuit-breaker.service.ts | 279 ++++++++++++++++++++++++ specs/003-non-calibre-books/tasks.md | 4 +- 3 files changed, 318 insertions(+), 2 deletions(-) create mode 100644 lib/providers/manual.provider.ts create mode 100644 lib/services/circuit-breaker.service.ts diff --git a/lib/providers/manual.provider.ts b/lib/providers/manual.provider.ts new file mode 100644 index 00000000..87571fe4 --- /dev/null +++ b/lib/providers/manual.provider.ts @@ -0,0 +1,37 @@ +/** + * Manual Provider + * + * Provider for manually-entered books (no external metadata source). + * Always enabled, no authentication, no external API calls. + * + * This provider exists primarily for consistency in the provider architecture, + * even though it doesn't fetch external metadata. + */ + +import type { + IMetadataProvider, + ProviderCapabilities, + ProviderHealth, + BookSource, +} from "./base/IMetadataProvider"; + +export class ManualProvider implements IMetadataProvider { + readonly id: BookSource = "manual"; + readonly name: string = "Manual Entry"; + readonly capabilities: ProviderCapabilities = { + hasSearch: false, + hasMetadataFetch: false, + hasSync: false, + requiresAuth: false, + }; + + /** + * Health check always returns healthy (no external dependency) + */ + async healthCheck(): Promise { + return "healthy"; + } +} + +// Export singleton instance +export const manualProvider = new ManualProvider(); diff --git a/lib/services/circuit-breaker.service.ts b/lib/services/circuit-breaker.service.ts new file mode 100644 index 00000000..09da9fb6 --- /dev/null +++ b/lib/services/circuit-breaker.service.ts @@ -0,0 +1,279 @@ +/** + * Circuit Breaker Service + * + * Implements the Circuit Breaker pattern for metadata providers to prevent + * cascading failures and provide automatic recovery. + * + * States: + * - CLOSED: Normal operation, requests pass through + * - OPEN: Failure threshold exceeded, requests fail fast + * - HALF_OPEN: Testing if service recovered + * + * See: specs/003-non-calibre-books/research.md (Decision 4: Circuit Breaker) + */ + +import { getLogger } from "@/lib/logger"; +import { providerConfigRepository } from "@/lib/repositories/provider-config.repository"; +import type { BookSource } from "@/lib/providers/base/IMetadataProvider"; + +const logger = getLogger().child({ module: "circuit-breaker" }); + +/** + * Circuit breaker configuration + */ +interface CircuitBreakerConfig { + /** Number of failures before opening circuit */ + failureThreshold: number; + /** Time in ms to wait before attempting recovery (OPEN → HALF_OPEN) */ + cooldownPeriod: number; + /** Number of successful requests needed to close circuit from HALF_OPEN */ + successThreshold: number; +} + +/** + * Default configuration + */ +const DEFAULT_CONFIG: CircuitBreakerConfig = { + failureThreshold: 5, + cooldownPeriod: 60000, // 60 seconds + successThreshold: 2, +}; + +/** + * Circuit state for each provider + */ +interface CircuitState { + state: "CLOSED" | "OPEN" | "HALF_OPEN"; + failureCount: number; + successCount: number; // Only used in HALF_OPEN + lastFailure: Date | null; + lastStateChange: Date; +} + +/** + * Circuit Breaker Service + * + * Manages circuit breaker state for all providers with automatic + * state transitions and failure tracking. + */ +export class CircuitBreakerService { + private circuits: Map = new Map(); + private config: CircuitBreakerConfig; + + constructor(config: Partial = {}) { + this.config = { ...DEFAULT_CONFIG, ...config }; + } + + /** + * Initialize circuit state for a provider + */ + private async initializeCircuit(provider: BookSource): Promise { + // Load persisted state from database + const providerConfig = await providerConfigRepository.findByProvider(provider); + + const state: CircuitState = { + state: providerConfig?.circuitState ?? "CLOSED", + failureCount: providerConfig?.failureCount ?? 0, + successCount: 0, + lastFailure: providerConfig?.lastFailure ?? null, + lastStateChange: new Date(), + }; + + this.circuits.set(provider, state); + return state; + } + + /** + * Get or initialize circuit state for a provider + */ + private async getCircuitState(provider: BookSource): Promise { + let state = this.circuits.get(provider); + if (!state) { + state = await this.initializeCircuit(provider); + } + + // Check if circuit should transition from OPEN to HALF_OPEN + if (state.state === "OPEN" && state.lastFailure) { + const timeSinceFailure = Date.now() - state.lastFailure.getTime(); + if (timeSinceFailure >= this.config.cooldownPeriod) { + await this.transitionTo(provider, "HALF_OPEN"); + state = this.circuits.get(provider)!; + } + } + + return state; + } + + /** + * Check if request can proceed through circuit + * + * @returns true if request should proceed, false if circuit is OPEN + */ + async canProceed(provider: BookSource): Promise { + const state = await this.getCircuitState(provider); + return state.state !== "OPEN"; + } + + /** + * Record successful request + */ + async recordSuccess(provider: BookSource): Promise { + const state = await this.getCircuitState(provider); + + if (state.state === "HALF_OPEN") { + state.successCount++; + logger.debug( + { + provider, + successCount: state.successCount, + threshold: this.config.successThreshold, + }, + "Circuit breaker success in HALF_OPEN state" + ); + + // Close circuit if success threshold met + if (state.successCount >= this.config.successThreshold) { + await this.transitionTo(provider, "CLOSED"); + } + } else if (state.state === "CLOSED" && state.failureCount > 0) { + // Reset failure count on success + state.failureCount = 0; + await providerConfigRepository.resetFailureCount(provider); + logger.debug({ provider }, "Reset failure count after success"); + } + } + + /** + * Record failed request + */ + async recordFailure(provider: BookSource): Promise { + const state = await this.getCircuitState(provider); + state.failureCount++; + state.lastFailure = new Date(); + + // Persist failure to database + await providerConfigRepository.updateCircuitState( + provider, + state.state, + state.failureCount, + state.lastFailure + ); + + logger.warn( + { + provider, + failureCount: state.failureCount, + threshold: this.config.failureThreshold, + currentState: state.state, + }, + "Circuit breaker recorded failure" + ); + + // Transition to OPEN if threshold exceeded + if ( + state.state === "CLOSED" && + state.failureCount >= this.config.failureThreshold + ) { + await this.transitionTo(provider, "OPEN"); + } else if (state.state === "HALF_OPEN") { + // Any failure in HALF_OPEN goes back to OPEN + await this.transitionTo(provider, "OPEN"); + } + } + + /** + * Transition circuit to new state + */ + private async transitionTo( + provider: BookSource, + newState: CircuitState["state"] + ): Promise { + const state = this.circuits.get(provider); + if (!state) { + await this.initializeCircuit(provider); + return; + } + + const oldState = state.state; + state.state = newState; + state.lastStateChange = new Date(); + state.successCount = 0; // Reset success count on state change + + // Persist state to database + await providerConfigRepository.updateCircuitState(provider, newState); + + // Update health status based on state + const healthStatus = newState === "OPEN" ? "unavailable" : "healthy"; + await providerConfigRepository.updateHealth( + provider, + healthStatus, + new Date() + ); + + logger.info( + { + provider, + oldState, + newState, + failureCount: state.failureCount, + }, + "Circuit breaker state transition" + ); + } + + /** + * Manually reset circuit to CLOSED state + * + * Use when you know the provider is healthy again + */ + async reset(provider: BookSource): Promise { + const state = this.circuits.get(provider); + if (state) { + state.failureCount = 0; + state.successCount = 0; + state.lastFailure = null; + await providerConfigRepository.resetFailureCount(provider); + } + + await this.transitionTo(provider, "CLOSED"); + logger.info({ provider }, "Circuit breaker manually reset"); + } + + /** + * Get current circuit state for a provider + */ + async getState(provider: BookSource): Promise { + const state = await this.getCircuitState(provider); + return state.state; + } + + /** + * Get circuit statistics for a provider + */ + async getStats(provider: BookSource): Promise<{ + state: CircuitState["state"]; + failureCount: number; + successCount: number; + lastFailure: Date | null; + timeUntilRetry: number | null; + }> { + const state = await this.getCircuitState(provider); + + let timeUntilRetry: number | null = null; + if (state.state === "OPEN" && state.lastFailure) { + const elapsed = Date.now() - state.lastFailure.getTime(); + timeUntilRetry = Math.max(0, this.config.cooldownPeriod - elapsed); + } + + return { + state: state.state, + failureCount: state.failureCount, + successCount: state.successCount, + lastFailure: state.lastFailure, + timeUntilRetry, + }; + } +} + +// Export singleton instance +export const circuitBreakerService = new CircuitBreakerService(); diff --git a/specs/003-non-calibre-books/tasks.md b/specs/003-non-calibre-books/tasks.md index 5ff6fa63..17e4a69a 100644 --- a/specs/003-non-calibre-books/tasks.md +++ b/specs/003-non-calibre-books/tasks.md @@ -65,13 +65,13 @@ This document breaks down the multi-source book tracking feature into executable ### Service Layer Foundation -- [ ] T013 [P] Create CircuitBreakerService with state machine (CLOSED/OPEN/HALF_OPEN) in lib/services/circuit-breaker.service.ts +- [X] T013 [P] Create CircuitBreakerService with state machine (CLOSED/OPEN/HALF_OPEN) in lib/services/circuit-breaker.service.ts - [ ] T014 [P] Implement ProviderService for provider orchestration in lib/services/provider.service.ts - [ ] T015 [P] Create MigrationService for source migration workflows in lib/services/migration.service.ts ### Provider Implementations (Stubs) -- [ ] T016 [P] Create ManualProvider stub (always enabled, no external API) in lib/providers/manual.provider.ts +- [X] T016 [P] Create ManualProvider stub (always enabled, no external API) in lib/providers/manual.provider.ts - [ ] T017 [P] Refactor existing Calibre sync to CalibreProvider implementation in lib/providers/calibre.provider.ts - [ ] T018 [P] Create HardcoverProvider stub with auth handling in lib/providers/hardcover.provider.ts - [ ] T019 [P] Create OpenLibraryProvider stub (public API, no auth) in lib/providers/openlibrary.provider.ts From ed408bb1e9471d7d6e5240d8eca949d9f829457a Mon Sep 17 00:00:00 2001 From: Mason Fox Date: Thu, 5 Feb 2026 18:34:45 -0500 Subject: [PATCH 07/89] feat(spec-003): Phase 2 complete - Service layer and provider stubs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Completed tasks T014-T020: Service Layer (T014-T015): - ProviderService: Orchestration layer with circuit breaker protection - Wraps search, fetchMetadata, sync operations - Health check coordination - Provider enable/disable at runtime - MigrationService: Source migration with pessimistic locking - Transactional updates with FOR UPDATE locking - Validation rules (manual → external only) - Metadata update during migration Provider Implementations (T016-T020): - ManualProvider: Stub for user-entered books (no external API) - CalibreProvider: Wraps existing syncCalibreLibrary functionality - Implements sync() and fetchMetadata() methods - Health check via database connectivity test - HardcoverProvider: Stub with search/fetch placeholders - OpenLibraryProvider: Stub with search/fetch placeholders - Provider registration: initializeProviders() function in ProviderRegistry Architecture: - All providers implement IMetadataProvider interface - Circuit breaker protects all provider operations - Health checks run independently per provider - Priority-based provider ordering (Calibre=1, Hardcover=10, OpenLibrary=20, Manual=99) Phase 2 Status: 20/20 tasks complete ✅ Next: Phase 3 (Manual Book Addition UI + Backend) --- lib/providers/base/ProviderRegistry.ts | 60 +++++ lib/providers/calibre.provider.ts | 153 +++++++++++ lib/providers/hardcover.provider.ts | 86 ++++++ lib/providers/openlibrary.provider.ts | 84 ++++++ lib/services/migration.service.ts | 359 +++++++++++++++++++++++++ lib/services/provider.service.ts | 355 ++++++++++++++++++++++++ specs/003-non-calibre-books/tasks.md | 12 +- 7 files changed, 1103 insertions(+), 6 deletions(-) create mode 100644 lib/providers/calibre.provider.ts create mode 100644 lib/providers/hardcover.provider.ts create mode 100644 lib/providers/openlibrary.provider.ts create mode 100644 lib/services/migration.service.ts create mode 100644 lib/services/provider.service.ts diff --git a/lib/providers/base/ProviderRegistry.ts b/lib/providers/base/ProviderRegistry.ts index e8a8667b..7cdd0b38 100644 --- a/lib/providers/base/ProviderRegistry.ts +++ b/lib/providers/base/ProviderRegistry.ts @@ -280,3 +280,63 @@ export class ProviderRegistry { return this.initialized; } } + +/** + * Initialize all providers + * + * Registers all available metadata providers with their default configurations. + * Should be called once at application startup. + * + * @example + * ```typescript + * // In app initialization + * initializeProviders(); + * ``` + */ +export function initializeProviders(): void { + // Skip if already initialized (prevent duplicate registration) + if (ProviderRegistry.isInitialized()) { + logger.debug("Provider registry already initialized - skipping"); + return; + } + + // Import providers lazily to avoid circular dependencies + const { calibreProvider } = require("@/lib/providers/calibre.provider"); + const { manualProvider } = require("@/lib/providers/manual.provider"); + const { hardcoverProvider } = require("@/lib/providers/hardcover.provider"); + const { openLibraryProvider } = require("@/lib/providers/openlibrary.provider"); + + // Register providers with priorities matching database seed data + // Priority: lower = higher priority (Calibre=1, Hardcover=10, OpenLibrary=20, Manual=99) + + ProviderRegistry.register(calibreProvider, { + enabled: true, + priority: 1, // Highest priority - primary source + }); + + ProviderRegistry.register(manualProvider, { + enabled: true, + priority: 99, // Lowest priority - fallback for user-entered books + }); + + ProviderRegistry.register(hardcoverProvider, { + enabled: true, + priority: 10, // Medium-high priority + }); + + ProviderRegistry.register(openLibraryProvider, { + enabled: true, + priority: 20, // Medium priority + }); + + // Mark as initialized to prevent re-registration + ProviderRegistry.markInitialized(); + + logger.info( + { + providers: ["calibre", "manual", "hardcover", "openlibrary"], + count: 4, + }, + "Provider registry initialized with 4 providers" + ); +} diff --git a/lib/providers/calibre.provider.ts b/lib/providers/calibre.provider.ts new file mode 100644 index 00000000..bba7f362 --- /dev/null +++ b/lib/providers/calibre.provider.ts @@ -0,0 +1,153 @@ +/** + * Calibre Provider + * + * Provider implementation for Calibre library integration. + * Wraps existing sync-service.ts functionality into the IMetadataProvider interface. + * + * See: specs/003-non-calibre-books/research.md (Decision 1: Provider Interface) + */ + +import { getLogger } from "@/lib/logger"; +import { syncCalibreLibrary } from "@/lib/sync-service"; +import type { + IMetadataProvider, + ProviderCapabilities, + ProviderHealth, + SyncResult as ProviderSyncResult, + BookMetadata, +} from "@/lib/providers/base/IMetadataProvider"; +import { getCalibreDB } from "@/lib/db/calibre"; +import { bookRepository } from "@/lib/repositories/book.repository"; + +const logger = getLogger().child({ module: "calibre-provider" }); + +/** + * Calibre Provider + * + * Integrates with local Calibre library database for book synchronization. + * + * Capabilities: + * - hasSearch: false (Calibre doesn't have search API, only full sync) + * - hasMetadataFetch: true (can fetch book metadata by calibreId) + * - hasSync: true (full library synchronization) + * - requiresAuth: false (local database access) + */ +class CalibreProvider implements IMetadataProvider { + readonly id = "calibre" as const; + readonly name = "Calibre Library"; + + readonly capabilities: ProviderCapabilities = { + hasSearch: false, // Calibre doesn't support search - only full sync + hasMetadataFetch: true, // Can fetch by calibreId + hasSync: true, // Full library sync + requiresAuth: false, // Local database + }; + + /** + * Fetch metadata for a Calibre book by calibreId + * + * @param externalId - Calibre book ID (as string) + * @returns Book metadata + * @throws Error if book not found or calibreId invalid + */ + async fetchMetadata(externalId: string): Promise { + const calibreId = parseInt(externalId, 10); + if (isNaN(calibreId)) { + throw new Error(`Invalid Calibre ID: ${externalId}`); + } + + logger.debug({ calibreId }, "Fetching Calibre book metadata"); + + // Fetch book from Tome database (which syncs from Calibre) + const book = await bookRepository.findByCalibreId(calibreId); + if (!book) { + throw new Error(`Book with Calibre ID ${calibreId} not found`); + } + + // Map to BookMetadata format + const metadata: BookMetadata = { + title: book.title, + authors: book.authors, + isbn: book.isbn ?? undefined, + description: book.description ?? undefined, + publisher: book.publisher ?? undefined, + pubDate: book.pubDate ?? undefined, + totalPages: book.totalPages ?? undefined, + series: book.series ?? undefined, + seriesIndex: book.seriesIndex ?? undefined, + externalId: String(calibreId), + tags: book.tags, + rating: book.rating ?? undefined, + }; + + logger.debug( + { calibreId, title: metadata.title }, + "Successfully fetched Calibre metadata" + ); + + return metadata; + } + + /** + * Sync entire Calibre library + * + * Delegates to existing syncCalibreLibrary() function. + * + * @returns Sync statistics + * @throws Error if sync fails + */ + async sync(): Promise { + logger.info("Starting Calibre library sync"); + + try { + const result = await syncCalibreLibrary(); + + if (!result.success) { + throw new Error(result.error || "Calibre sync failed"); + } + + logger.info( + { + added: result.syncedCount, + updated: result.updatedCount, + removed: result.removedCount, + }, + "Calibre library sync completed" + ); + + return { + added: result.syncedCount, + updated: result.updatedCount, + removed: result.removedCount, + errors: 0, + }; + } catch (error) { + const err = error instanceof Error ? error : new Error(String(error)); + logger.error({ error: err.message }, "Calibre library sync failed"); + throw err; + } + } + + /** + * Check Calibre database connectivity + * + * @returns "healthy" if connection succeeds, "unavailable" otherwise + */ + async healthCheck(): Promise { + try { + const calibreDb = getCalibreDB(); + // Test with a simple query + const result = calibreDb.prepare("SELECT 1 as test").get() as { test: number }; + return result.test === 1 ? "healthy" : "unavailable"; + } catch (error) { + logger.warn( + { error: error instanceof Error ? error.message : String(error) }, + "Calibre health check failed" + ); + return "unavailable"; + } + } +} + +// Export singleton instance +export const calibreProvider = new CalibreProvider(); diff --git a/lib/providers/hardcover.provider.ts b/lib/providers/hardcover.provider.ts new file mode 100644 index 00000000..45a6b480 --- /dev/null +++ b/lib/providers/hardcover.provider.ts @@ -0,0 +1,86 @@ +/** + * Hardcover Provider (Stub) + * + * Provider implementation for Hardcover.app API integration. + * Currently a stub - full implementation to be completed in Phase 7. + * + * See: specs/003-non-calibre-books/spec.md (User Story 4) + */ + +import { getLogger } from "@/lib/logger"; +import type { + IMetadataProvider, + ProviderCapabilities, + ProviderHealth, + SearchResult, + BookMetadata, +} from "@/lib/providers/base/IMetadataProvider"; + +const logger = getLogger().child({ module: "hardcover-provider" }); + +/** + * Hardcover Provider (Stub) + * + * Stub implementation for Hardcover.app integration. + * + * Capabilities (future): + * - hasSearch: true (search Hardcover catalog) + * - hasMetadataFetch: true (fetch book details by Hardcover ID) + * - hasSync: false (no bulk sync - manual/search-based only) + * - requiresAuth: false (public API, no auth required initially) + * + * Note: requiresAuth may become true if we need API keys for higher rate limits + */ +class HardcoverProvider implements IMetadataProvider { + readonly id = "hardcover" as const; + readonly name = "Hardcover.app"; + + readonly capabilities: ProviderCapabilities = { + hasSearch: true, + hasMetadataFetch: true, + hasSync: false, // No bulk sync - users search and select individual books + requiresAuth: false, // May change to true if API keys needed + }; + + /** + * Search Hardcover catalog (NOT IMPLEMENTED) + * + * @throws Error - Not implemented in stub + */ + async search(query: string): Promise { + logger.warn({ query }, "Hardcover search called but not yet implemented"); + throw new Error( + "Hardcover search not implemented - placeholder for Phase 7 (User Story 4)" + ); + } + + /** + * Fetch book metadata from Hardcover (NOT IMPLEMENTED) + * + * @throws Error - Not implemented in stub + */ + async fetchMetadata(externalId: string): Promise { + logger.warn( + { externalId }, + "Hardcover fetchMetadata called but not yet implemented" + ); + throw new Error( + "Hardcover fetchMetadata not implemented - placeholder for Phase 7 (User Story 4)" + ); + } + + /** + * Health check (stub - always healthy) + * + * In full implementation, this would check Hardcover API availability. + * + * @returns "healthy" (stub always returns healthy) + */ + async healthCheck(): Promise { + logger.debug("Hardcover health check (stub - always healthy)"); + return "healthy"; + } +} + +// Export singleton instance +export const hardcoverProvider = new HardcoverProvider(); diff --git a/lib/providers/openlibrary.provider.ts b/lib/providers/openlibrary.provider.ts new file mode 100644 index 00000000..14e94745 --- /dev/null +++ b/lib/providers/openlibrary.provider.ts @@ -0,0 +1,84 @@ +/** + * OpenLibrary Provider (Stub) + * + * Provider implementation for OpenLibrary.org API integration. + * Currently a stub - full implementation to be completed in Phase 7. + * + * See: specs/003-non-calibre-books/spec.md (User Story 4) + */ + +import { getLogger } from "@/lib/logger"; +import type { + IMetadataProvider, + ProviderCapabilities, + ProviderHealth, + SearchResult, + BookMetadata, +} from "@/lib/providers/base/IMetadataProvider"; + +const logger = getLogger().child({ module: "openlibrary-provider" }); + +/** + * OpenLibrary Provider (Stub) + * + * Stub implementation for OpenLibrary.org integration. + * + * Capabilities (future): + * - hasSearch: true (search OpenLibrary catalog) + * - hasMetadataFetch: true (fetch book details by OpenLibrary ID) + * - hasSync: false (no bulk sync - manual/search-based only) + * - requiresAuth: false (public API, no authentication required) + */ +class OpenLibraryProvider implements IMetadataProvider { + readonly id = "openlibrary" as const; + readonly name = "OpenLibrary"; + + readonly capabilities: ProviderCapabilities = { + hasSearch: true, + hasMetadataFetch: true, + hasSync: false, // No bulk sync - users search and select individual books + requiresAuth: false, // Public API + }; + + /** + * Search OpenLibrary catalog (NOT IMPLEMENTED) + * + * @throws Error - Not implemented in stub + */ + async search(query: string): Promise { + logger.warn({ query }, "OpenLibrary search called but not yet implemented"); + throw new Error( + "OpenLibrary search not implemented - placeholder for Phase 7 (User Story 4)" + ); + } + + /** + * Fetch book metadata from OpenLibrary (NOT IMPLEMENTED) + * + * @throws Error - Not implemented in stub + */ + async fetchMetadata(externalId: string): Promise { + logger.warn( + { externalId }, + "OpenLibrary fetchMetadata called but not yet implemented" + ); + throw new Error( + "OpenLibrary fetchMetadata not implemented - placeholder for Phase 7 (User Story 4)" + ); + } + + /** + * Health check (stub - always healthy) + * + * In full implementation, this would check OpenLibrary API availability. + * + * @returns "healthy" (stub always returns healthy) + */ + async healthCheck(): Promise { + logger.debug("OpenLibrary health check (stub - always healthy)"); + return "healthy"; + } +} + +// Export singleton instance +export const openLibraryProvider = new OpenLibraryProvider(); diff --git a/lib/services/migration.service.ts b/lib/services/migration.service.ts new file mode 100644 index 00000000..32011403 --- /dev/null +++ b/lib/services/migration.service.ts @@ -0,0 +1,359 @@ +/** + * Migration Service + * + * Handles source migration workflows (e.g., manual → calibre, manual → hardcover). + * Provides transactional updates with pessimistic locking to prevent race conditions. + * + * Migration Rules: + * - Only manual → external provider migrations allowed + * - Cross-provider migrations (calibre ↔ hardcover) not allowed + * - Target must have externalId (cannot migrate to manual source) + * + * See: specs/003-non-calibre-books/spec.md (User Story 5, FR-016) + */ + +import { getLogger } from "@/lib/logger"; +import { db } from "@/lib/db/sqlite"; +import { sqlite } from "@/lib/db/sqlite"; +import { books } from "@/lib/db/schema/books"; +import { eq } from "drizzle-orm"; +import type { BookSource } from "@/lib/providers/base/IMetadataProvider"; + +const logger = getLogger().child({ module: "migration-service" }); + +/** + * Migration validation error + */ +export class MigrationError extends Error { + constructor( + public readonly bookId: number, + public readonly reason: string + ) { + super(`Migration failed for book ${bookId}: ${reason}`); + this.name = "MigrationError"; + } +} + +/** + * Migration options + */ +export interface MigrateSourceOptions { + /** Book ID to migrate */ + bookId: number; + + /** Target source provider */ + targetSource: BookSource; + + /** External ID for the target provider */ + targetExternalId: string; + + /** Optional: Updated metadata to apply during migration */ + metadata?: { + title?: string; + authors?: string[]; + isbn?: string; + description?: string; + publisher?: string; + pubDate?: Date; + totalPages?: number; + series?: string; + seriesIndex?: number; + tags?: string[]; + coverImageUrl?: string; + }; +} + +/** + * Migration result + */ +export interface MigrationResult { + bookId: number; + oldSource: BookSource; + newSource: BookSource; + oldExternalId: string | null; + newExternalId: string; + updatedFields: string[]; +} + +/** + * Migration Service + * + * Orchestrates book source migrations with validation, locking, and logging. + * + * @example + * ```typescript + * // Migrate manual book to Hardcover + * const result = await migrationService.migrateSource({ + * bookId: 123, + * targetSource: 'hardcover', + * targetExternalId: 'hc_abc123', + * metadata: { description: 'New description from Hardcover' } + * }); + * ``` + */ +export class MigrationService { + + /** + * Validate migration request + * + * @throws MigrationError if migration is invalid + */ + private validateMigration( + bookId: number, + currentSource: BookSource, + targetSource: BookSource, + targetExternalId: string + ): void { + // Rule: Cannot migrate to manual source + if (targetSource === "manual") { + throw new MigrationError( + bookId, + "Cannot migrate to 'manual' source - manual books have no externalId" + ); + } + + // Rule: Target must have externalId + if (!targetExternalId || targetExternalId.trim() === "") { + throw new MigrationError( + bookId, + "Target externalId is required for migration" + ); + } + + // Rule: Only manual → external migrations allowed + if (currentSource !== "manual") { + throw new MigrationError( + bookId, + `Only manual → external migrations allowed (current source: ${currentSource})` + ); + } + + // Rule: Cannot migrate to same source (string comparison) + if (String(currentSource) === String(targetSource)) { + throw new MigrationError( + bookId, + `Book is already sourced from ${targetSource}` + ); + } + + logger.debug( + { bookId, currentSource, targetSource, targetExternalId }, + "Migration validation passed" + ); + } + + /** + * Migrate book source with transactional safety + * + * Uses pessimistic locking (SELECT ... FOR UPDATE) to prevent concurrent + * modifications during migration. + * + * @throws MigrationError if migration invalid or book not found + * @throws Error if database transaction fails + */ + async migrateSource(options: MigrateSourceOptions): Promise { + const { bookId, targetSource, targetExternalId, metadata } = options; + + logger.info( + { bookId, targetSource, targetExternalId }, + "Starting source migration" + ); + + return new Promise((resolve, reject) => { + const transaction = sqlite.transaction(() => { + // Step 1: Lock row with FOR UPDATE (pessimistic locking) + const book = sqlite + .prepare( + ` + SELECT id, source, externalId, title, authors, isbn, description, + publisher, pubDate, totalPages, series, seriesIndex, tags + FROM books + WHERE id = ? + FOR UPDATE + ` + ) + .get(bookId) as any; + + if (!book) { + throw new MigrationError(bookId, "Book not found"); + } + + const currentSource = book.source as BookSource; + const currentExternalId = book.externalId as string | null; + + // Step 2: Validate migration + try { + this.validateMigration( + bookId, + currentSource, + targetSource, + targetExternalId + ); + } catch (error) { + // Re-throw MigrationError to be caught by outer try/catch + throw error; + } + + // Step 3: Build update statement + const updates: Record = { + source: targetSource, + externalId: targetExternalId, + }; + + const updatedFields = ["source", "externalId"]; + + // Apply optional metadata updates + if (metadata) { + if (metadata.title !== undefined) { + updates.title = metadata.title; + updatedFields.push("title"); + } + if (metadata.authors !== undefined) { + updates.authors = metadata.authors; + updatedFields.push("authors"); + } + if (metadata.isbn !== undefined) { + updates.isbn = metadata.isbn; + updatedFields.push("isbn"); + } + if (metadata.description !== undefined) { + updates.description = metadata.description; + updatedFields.push("description"); + } + if (metadata.publisher !== undefined) { + updates.publisher = metadata.publisher; + updatedFields.push("publisher"); + } + if (metadata.pubDate !== undefined) { + updates.pubDate = metadata.pubDate; + updatedFields.push("pubDate"); + } + if (metadata.totalPages !== undefined) { + updates.totalPages = metadata.totalPages; + updatedFields.push("totalPages"); + } + if (metadata.series !== undefined) { + updates.series = metadata.series; + updatedFields.push("series"); + } + if (metadata.seriesIndex !== undefined) { + updates.seriesIndex = metadata.seriesIndex; + updatedFields.push("seriesIndex"); + } + if (metadata.tags !== undefined) { + updates.tags = metadata.tags; + updatedFields.push("tags"); + } + if (metadata.coverImageUrl !== undefined) { + updates.coverImageUrl = metadata.coverImageUrl; + updatedFields.push("coverImageUrl"); + } + } + + // Step 4: Execute update + const setClauses = Object.keys(updates).map((key) => `${key} = ?`); + const values = Object.values(updates); + + const updateSql = ` + UPDATE books + SET ${setClauses.join(", ")}, updatedAt = CURRENT_TIMESTAMP + WHERE id = ? + `; + + sqlite.prepare(updateSql).run(...values, bookId); + + logger.info( + { + bookId, + oldSource: currentSource, + newSource: targetSource, + oldExternalId: currentExternalId, + newExternalId: targetExternalId, + updatedFields, + }, + "Source migration completed successfully" + ); + + return { + bookId, + oldSource: currentSource, + newSource: targetSource, + oldExternalId: currentExternalId, + newExternalId: targetExternalId, + updatedFields, + }; + }); + + try { + const result = transaction(); + resolve(result); + } catch (error) { + const err = error instanceof Error ? error : new Error(String(error)); + logger.error( + { bookId, targetSource, error: err.message }, + "Source migration failed" + ); + reject(err); + } + }); + } + + /** + * Check if migration is allowed for a book + * + * Non-throwing validation check for UI pre-flight. + * + * @returns Object with allowed flag and reason if not allowed + */ + async canMigrate( + bookId: number, + targetSource: BookSource + ): Promise<{ allowed: boolean; reason?: string }> { + const book = sqlite + .prepare("SELECT source FROM books WHERE id = ?") + .get(bookId) as { source: BookSource } | undefined; + + if (!book) { + return { allowed: false, reason: "Book not found" }; + } + + // Rule: Cannot migrate to manual + if (targetSource === "manual") { + return { allowed: false, reason: "Cannot migrate to manual source" }; + } + + // Rule: Only manual → external + if (book.source !== "manual") { + return { + allowed: false, + reason: `Only manual books can be migrated (current source: ${book.source})`, + }; + } + + // Rule: Cannot migrate to same source (string comparison) + if (String(book.source) === String(targetSource)) { + return { + allowed: false, + reason: `Book is already sourced from ${targetSource}`, + }; + } + + return { allowed: true }; + } + + /** + * Get migration history for a book + * + * Note: Currently we don't track migration history in a separate table. + * This could be added in a future enhancement. + * + * @returns Empty array (placeholder for future enhancement) + */ + async getMigrationHistory(bookId: number): Promise { + logger.debug({ bookId }, "Migration history requested (not yet implemented)"); + return []; + } +} + +// Export singleton instance +export const migrationService = new MigrationService(); diff --git a/lib/services/provider.service.ts b/lib/services/provider.service.ts new file mode 100644 index 00000000..d5035bb3 --- /dev/null +++ b/lib/services/provider.service.ts @@ -0,0 +1,355 @@ +/** + * Provider Service + * + * Orchestration layer for metadata providers. Wraps provider operations with + * circuit breaker protection, health checks, and error handling. + * + * See: specs/003-non-calibre-books/research.md (Decision 4: Circuit Breaker) + */ + +import { getLogger } from "@/lib/logger"; +import { ProviderRegistry } from "@/lib/providers/base/ProviderRegistry"; +import { circuitBreakerService } from "@/lib/services/circuit-breaker.service"; +import { providerConfigRepository } from "@/lib/repositories/provider-config.repository"; +import type { + IMetadataProvider, + BookSource, + SearchResult, + BookMetadata, + SyncResult, + ProviderHealth, +} from "@/lib/providers/base/IMetadataProvider"; + +const logger = getLogger().child({ module: "provider-service" }); + +/** + * Provider operation error + */ +export class ProviderError extends Error { + constructor( + public readonly provider: BookSource, + public readonly operation: string, + message: string, + public readonly cause?: Error + ) { + super(`Provider '${provider}' ${operation} failed: ${message}`); + this.name = "ProviderError"; + } +} + +/** + * Circuit open error (provider unavailable) + */ +export class CircuitOpenError extends ProviderError { + constructor(provider: BookSource, operation: string) { + super( + provider, + operation, + "Circuit breaker is OPEN - provider unavailable" + ); + this.name = "CircuitOpenError"; + } +} + +/** + * Provider Service + * + * Orchestrates metadata provider operations with circuit breaker protection, + * health monitoring, and automatic failover. + * + * @example + * ```typescript + * // Search across all enabled providers + * const results = await providerService.search('calibre', 'The Great Gatsby'); + * + * // Fetch metadata with circuit breaker protection + * const metadata = await providerService.fetchMetadata('hardcover', 'abc123'); + * + * // Run health checks + * await providerService.healthCheckAll(); + * ``` + */ +export class ProviderService { + /** + * Get provider by ID + * + * @throws Error if provider not found + */ + getProvider(source: BookSource): IMetadataProvider { + const provider = ProviderRegistry.get(source); + if (!provider) { + throw new Error(`Provider '${source}' not found`); + } + return provider; + } + + /** + * Get all enabled providers sorted by priority + */ + getEnabledProviders(): IMetadataProvider[] { + return ProviderRegistry.getEnabled(); + } + + /** + * Get providers with specific capability + */ + getProvidersByCapability( + capability: keyof IMetadataProvider["capabilities"] + ): IMetadataProvider[] { + return ProviderRegistry.getByCapability(capability); + } + + /** + * Search for books using a specific provider + * + * @throws CircuitOpenError if circuit breaker is open + * @throws ProviderError if search fails + */ + async search(source: BookSource, query: string): Promise { + const provider = this.getProvider(source); + + // Validate capability + if (!provider.capabilities.hasSearch) { + throw new ProviderError( + source, + "search", + "Provider does not support search" + ); + } + + // Check circuit breaker + const canProceed = await circuitBreakerService.canProceed(source); + if (!canProceed) { + throw new CircuitOpenError(source, "search"); + } + + // Execute search with circuit breaker tracking + try { + logger.debug({ provider: source, query }, "Executing provider search"); + const results = await provider.search!(query); + await circuitBreakerService.recordSuccess(source); + logger.info( + { provider: source, query, resultCount: results.length }, + "Provider search succeeded" + ); + return results; + } catch (error) { + await circuitBreakerService.recordFailure(source); + const err = error instanceof Error ? error : new Error(String(error)); + logger.error( + { provider: source, query, error: err.message }, + "Provider search failed" + ); + throw new ProviderError(source, "search", err.message, err); + } + } + + /** + * Fetch full metadata using a specific provider + * + * @throws CircuitOpenError if circuit breaker is open + * @throws ProviderError if fetch fails + */ + async fetchMetadata( + source: BookSource, + externalId: string + ): Promise { + const provider = this.getProvider(source); + + // Validate capability + if (!provider.capabilities.hasMetadataFetch) { + throw new ProviderError( + source, + "fetchMetadata", + "Provider does not support metadata fetch" + ); + } + + // Check circuit breaker + const canProceed = await circuitBreakerService.canProceed(source); + if (!canProceed) { + throw new CircuitOpenError(source, "fetchMetadata"); + } + + // Execute fetch with circuit breaker tracking + try { + logger.debug( + { provider: source, externalId }, + "Executing provider metadata fetch" + ); + const metadata = await provider.fetchMetadata!(externalId); + await circuitBreakerService.recordSuccess(source); + logger.info( + { provider: source, externalId, title: metadata.title }, + "Provider metadata fetch succeeded" + ); + return metadata; + } catch (error) { + await circuitBreakerService.recordFailure(source); + const err = error instanceof Error ? error : new Error(String(error)); + logger.error( + { provider: source, externalId, error: err.message }, + "Provider metadata fetch failed" + ); + throw new ProviderError(source, "fetchMetadata", err.message, err); + } + } + + /** + * Sync entire library using a specific provider + * + * @throws CircuitOpenError if circuit breaker is open + * @throws ProviderError if sync fails + */ + async sync(source: BookSource): Promise { + const provider = this.getProvider(source); + + // Validate capability + if (!provider.capabilities.hasSync) { + throw new ProviderError( + source, + "sync", + "Provider does not support sync" + ); + } + + // Check circuit breaker + const canProceed = await circuitBreakerService.canProceed(source); + if (!canProceed) { + throw new CircuitOpenError(source, "sync"); + } + + // Execute sync with circuit breaker tracking + try { + logger.info({ provider: source }, "Starting provider sync"); + const result = await provider.sync!(); + await circuitBreakerService.recordSuccess(source); + logger.info( + { provider: source, result }, + "Provider sync completed successfully" + ); + return result; + } catch (error) { + await circuitBreakerService.recordFailure(source); + const err = error instanceof Error ? error : new Error(String(error)); + logger.error( + { provider: source, error: err.message }, + "Provider sync failed" + ); + throw new ProviderError(source, "sync", err.message, err); + } + } + + /** + * Run health check on a specific provider + * + * Updates provider health status in database. + * + * @returns Current health status + */ + async healthCheck(source: BookSource): Promise { + const provider = this.getProvider(source); + + try { + logger.debug({ provider: source }, "Running provider health check"); + const status = await provider.healthCheck(); + + // Update database + await providerConfigRepository.updateHealth(source, status, new Date()); + + // Update registry + ProviderRegistry.updateHealth(source, status); + + logger.debug( + { provider: source, status }, + "Provider health check completed" + ); + return status; + } catch (error) { + const err = error instanceof Error ? error : new Error(String(error)); + logger.error( + { provider: source, error: err.message }, + "Provider health check failed" + ); + + // Mark as unavailable + await providerConfigRepository.updateHealth( + source, + "unavailable", + new Date() + ); + ProviderRegistry.updateHealth(source, "unavailable"); + + return "unavailable"; + } + } + + /** + * Run health checks on all registered providers + * + * @returns Map of provider ID to health status + */ + async healthCheckAll(): Promise> { + const providers = ProviderRegistry.getAll(); + const results = new Map(); + + logger.info( + { providerCount: providers.length }, + "Running health checks on all providers" + ); + + // Run all health checks in parallel + await Promise.all( + providers.map(async (provider) => { + const status = await this.healthCheck(provider.id); + results.set(provider.id, status); + }) + ); + + const healthySummary = Array.from(results.entries()).reduce( + (acc, [id, status]) => { + acc[status] = (acc[status] || 0) + 1; + return acc; + }, + {} as Record + ); + + logger.info( + { results: healthySummary }, + "Completed health checks on all providers" + ); + + return results; + } + + /** + * Enable or disable a provider at runtime + * + * Updates both database and registry. + */ + async setEnabled(source: BookSource, enabled: boolean): Promise { + await providerConfigRepository.setEnabled(source, enabled); + ProviderRegistry.setEnabled(source, enabled); + logger.info({ provider: source, enabled }, "Updated provider enabled state"); + } + + /** + * Get circuit breaker statistics for a provider + */ + async getCircuitStats(source: BookSource) { + return circuitBreakerService.getStats(source); + } + + /** + * Manually reset circuit breaker for a provider + * + * Use when you know the provider is healthy again. + */ + async resetCircuit(source: BookSource): Promise { + await circuitBreakerService.reset(source); + logger.info({ provider: source }, "Circuit breaker manually reset"); + } +} + +// Export singleton instance +export const providerService = new ProviderService(); diff --git a/specs/003-non-calibre-books/tasks.md b/specs/003-non-calibre-books/tasks.md index 17e4a69a..0869af2f 100644 --- a/specs/003-non-calibre-books/tasks.md +++ b/specs/003-non-calibre-books/tasks.md @@ -66,16 +66,16 @@ This document breaks down the multi-source book tracking feature into executable ### Service Layer Foundation - [X] T013 [P] Create CircuitBreakerService with state machine (CLOSED/OPEN/HALF_OPEN) in lib/services/circuit-breaker.service.ts -- [ ] T014 [P] Implement ProviderService for provider orchestration in lib/services/provider.service.ts -- [ ] T015 [P] Create MigrationService for source migration workflows in lib/services/migration.service.ts +- [X] T014 [P] Implement ProviderService for provider orchestration in lib/services/provider.service.ts +- [X] T015 [P] Create MigrationService for source migration workflows in lib/services/migration.service.ts ### Provider Implementations (Stubs) - [X] T016 [P] Create ManualProvider stub (always enabled, no external API) in lib/providers/manual.provider.ts -- [ ] T017 [P] Refactor existing Calibre sync to CalibreProvider implementation in lib/providers/calibre.provider.ts -- [ ] T018 [P] Create HardcoverProvider stub with auth handling in lib/providers/hardcover.provider.ts -- [ ] T019 [P] Create OpenLibraryProvider stub (public API, no auth) in lib/providers/openlibrary.provider.ts -- [ ] T020 Register all providers in ProviderRegistry singleton in lib/providers/base/ProviderRegistry.ts +- [X] T017 [P] Refactor existing Calibre sync to CalibreProvider implementation in lib/providers/calibre.provider.ts +- [X] T018 [P] Create HardcoverProvider stub with auth handling in lib/providers/hardcover.provider.ts +- [X] T019 [P] Create OpenLibraryProvider stub (public API, no auth) in lib/providers/openlibrary.provider.ts +- [X] T020 Register all providers in ProviderRegistry singleton in lib/providers/base/ProviderRegistry.ts --- From 4b6d9c166ea4592dbd2aa2c7aac06447fa3e95eb Mon Sep 17 00:00:00 2001 From: Mason Fox Date: Thu, 5 Feb 2026 18:44:50 -0500 Subject: [PATCH 08/89] feat(spec-003): Phase 3 Backend - Manual book creation (T021-T027) Completed backend tasks for manual book addition: Validation (T022, T025, T027): - Created manual-book.schema.ts with Zod validation - Required: title, authors (1+ non-empty strings) - Optional: ISBN, publisher, pubDate, totalPages (1-10000), series, seriesIndex, tags - Comprehensive validation rules with user-friendly error messages Duplicate Detection (T023, T024): - Created duplicate-detection.service.ts with Levenshtein algorithm - 85% similarity threshold for title matching - Author matching with normalization - Returns potential duplicates sorted by similarity - Warning-only (doesn't prevent creation) Book Service Extensions (T022, T024): - Added createManualBook() method to bookService - Validates input using Zod schema - Checks for duplicates before creation - Creates book with source='manual', externalId=null, calibreId=null - Auto-creates initial 'to-read' session - Returns book + duplicate detection result - Added checkForDuplicates() for real-time UI checks - Fixed syncRatingToCalibre() to skip sync for non-Calibre books API Routes (T021, T026): - Extended POST /api/books for manual book creation - Detects manual creation vs legacy calibreId updates - Returns 201 Created with book + duplicates - Returns 400 with validation errors on invalid input - Created POST /api/books/validate endpoint - Real-time validation without creating book - Returns validation errors and duplicate detection results Next: Frontend components (ManualBookForm, DuplicateWarning, ProviderBadge) --- app/api/books/route.ts | 45 +++- app/api/books/validate/route.ts | 49 ++++ lib/services/book.service.ts | 137 +++++++++- lib/services/duplicate-detection.service.ts | 278 ++++++++++++++++++++ lib/validation/manual-book.schema.ts | 106 ++++++++ 5 files changed, 610 insertions(+), 5 deletions(-) create mode 100644 app/api/books/validate/route.ts create mode 100644 lib/services/duplicate-detection.service.ts create mode 100644 lib/validation/manual-book.schema.ts diff --git a/app/api/books/route.ts b/app/api/books/route.ts index a165d3a9..45855982 100644 --- a/app/api/books/route.ts +++ b/app/api/books/route.ts @@ -1,6 +1,8 @@ import { getLogger } from "@/lib/logger"; import { NextRequest, NextResponse } from "next/server"; import { bookRepository } from "@/lib/repositories"; +import { bookService } from "@/lib/services/book.service"; +import { ZodError } from "zod"; export const dynamic = 'force-dynamic'; @@ -66,10 +68,44 @@ export async function GET(request: NextRequest) { export async function POST(request: NextRequest) { try { const body = await request.json(); + + // Check if this is a manual book creation (has title field) + if (body.title && body.authors) { + // Manual book creation + try { + const result = await bookService.createManualBook(body); + + return NextResponse.json({ + book: result.book, + duplicates: result.duplicates, + }, { status: 201 }); + } catch (error) { + if (error instanceof ZodError) { + return NextResponse.json( + { + error: "Validation failed", + details: error.issues, + }, + { status: 400 } + ); + } + + getLogger().error({ err: error }, "Error creating manual book"); + return NextResponse.json( + { error: "Failed to create manual book" }, + { status: 500 } + ); + } + } + + // Legacy: Update book by calibreId const { calibreId, totalPages } = body; if (!calibreId) { - return NextResponse.json({ error: "calibreId is required" }, { status: 400 }); + return NextResponse.json( + { error: "Either (title + authors) or calibreId is required" }, + { status: 400 } + ); } const book = await bookRepository.findByCalibreId(calibreId); @@ -85,7 +121,10 @@ export async function POST(request: NextRequest) { return NextResponse.json(book); } catch (error) { - getLogger().error({ err: error }, "Error updating book"); - return NextResponse.json({ error: "Failed to update book" }, { status: 500 }); + getLogger().error({ err: error }, "Error in POST /api/books"); + return NextResponse.json( + { error: "Failed to process request" }, + { status: 500 } + ); } } diff --git a/app/api/books/validate/route.ts b/app/api/books/validate/route.ts new file mode 100644 index 00000000..cce6d99c --- /dev/null +++ b/app/api/books/validate/route.ts @@ -0,0 +1,49 @@ +/** + * POST /api/books/validate + * + * Real-time validation endpoint for manual book input. + * Returns validation errors without creating the book. + */ + +import { getLogger } from "@/lib/logger"; +import { NextRequest, NextResponse } from "next/server"; +import { validateManualBookInputSafe } from "@/lib/validation/manual-book.schema"; +import { bookService } from "@/lib/services/book.service"; + +export const dynamic = 'force-dynamic'; + +export async function POST(request: NextRequest) { + try { + const body = await request.json(); + + // Validate input + const validation = validateManualBookInputSafe(body); + + if (!validation.success) { + return NextResponse.json( + { + valid: false, + errors: validation.errors.issues, + }, + { status: 200 } // 200 OK with validation errors + ); + } + + // Check for duplicates + const duplicates = await bookService.checkForDuplicates( + validation.data.title, + validation.data.authors + ); + + return NextResponse.json({ + valid: true, + duplicates: duplicates, + }); + } catch (error) { + getLogger().error({ err: error }, "Error validating manual book input"); + return NextResponse.json( + { error: "Failed to validate input" }, + { status: 500 } + ); + } +} diff --git a/lib/services/book.service.ts b/lib/services/book.service.ts index 5f80bca3..1d3cdd1e 100644 --- a/lib/services/book.service.ts +++ b/lib/services/book.service.ts @@ -1,10 +1,13 @@ import { bookRepository, sessionRepository, progressRepository } from "@/lib/repositories"; -import type { Book } from "@/lib/db/schema/books"; +import type { Book, NewBook } from "@/lib/db/schema/books"; import type { ReadingSession } from "@/lib/db/schema/reading-sessions"; import type { ProgressLog } from "@/lib/db/schema/progress-logs"; import type { BookFilter } from "@/lib/repositories/book.repository"; import type { ICalibreService } from "@/lib/services/calibre.service"; import { getLogger } from "@/lib/logger"; +import { validateManualBookInput, type ManualBookInput } from "@/lib/validation/manual-book.schema"; +import { detectDuplicates, type DuplicateDetectionResult } from "@/lib/services/duplicate-detection.service"; +import { generateAuthorSort } from "@/lib/utils/author-sort"; /** * Book with enriched details (session, progress, read count) @@ -17,6 +20,14 @@ export interface BookWithDetails extends Book { totalReads: number; } +/** + * Manual book creation result with duplicate warnings + */ +export interface ManualBookCreationResult { + book: Book; + duplicates: DuplicateDetectionResult; +} + /** * BookService - Handles book CRUD operations and metadata updates * @@ -257,10 +268,132 @@ export class BookService { }; } + /** + * Create a manual book entry + * + * Creates a new book with source='manual' and performs duplicate detection. + * Validates input and automatically creates an initial reading session. + * + * @param input - Manual book data (title, authors, optional metadata) + * @returns Promise resolving to created book and duplicate detection result + * @throws {Error} If validation fails or book creation fails + * + * @example + * const result = await bookService.createManualBook({ + * title: 'The Great Gatsby', + * authors: ['F. Scott Fitzgerald'], + * totalPages: 180 + * }); + * + * if (result.duplicates.hasDuplicates) { + * console.log('Potential duplicates:', result.duplicates.duplicates); + * } + */ + async createManualBook(input: ManualBookInput): Promise { + const logger = getLogger(); + + // Validate input + const validatedInput = validateManualBookInput(input); + logger.debug({ title: validatedInput.title }, "Creating manual book"); + + // Check for duplicates (warning only, doesn't prevent creation) + const duplicates = await detectDuplicates( + validatedInput.title, + validatedInput.authors + ); + + if (duplicates.hasDuplicates) { + logger.info( + { + title: validatedInput.title, + duplicateCount: duplicates.duplicates.length, + }, + "Manual book creation detected potential duplicates" + ); + } + + // Prepare book data + const newBook: NewBook = { + // Required fields + title: validatedInput.title, + authors: validatedInput.authors, + authorSort: generateAuthorSort(validatedInput.authors), + + // Source fields + source: "manual", + externalId: null, + calibreId: null, + + // Optional metadata + isbn: validatedInput.isbn ?? null, + description: validatedInput.description ?? null, + publisher: validatedInput.publisher ?? null, + pubDate: validatedInput.pubDate ?? null, + totalPages: validatedInput.totalPages ?? null, + series: validatedInput.series ?? null, + seriesIndex: validatedInput.seriesIndex ?? null, + tags: validatedInput.tags ?? [], + + // Calibre-specific fields (null for manual books) + path: null, + lastSynced: null, + + // Default values + rating: null, + orphanedAt: null, + }; + + // Create book + const createdBook = await bookRepository.create(newBook); + + // Create initial reading session + await sessionRepository.create({ + bookId: createdBook.id, + status: "to-read", + startedDate: new Date().toISOString(), + sessionNumber: 1, + }); + + logger.info( + { + bookId: createdBook.id, + title: createdBook.title, + hasDuplicates: duplicates.hasDuplicates, + }, + "Manual book created successfully" + ); + + return { + book: createdBook, + duplicates, + }; + } + + /** + * Check for duplicate books (preview without creating) + * + * Used for real-time duplicate detection in the UI before submission. + * + * @param title - Book title to check + * @param authors - Book authors to check + * @returns Promise resolving to duplicate detection result + */ + async checkForDuplicates( + title: string, + authors: string[] + ): Promise { + return detectDuplicates(title, authors); + } + /** * Sync rating to Calibre (best effort) */ - private async syncRatingToCalibre(calibreId: number, rating: number | null): Promise { + private async syncRatingToCalibre(calibreId: number | null, rating: number | null): Promise { + // Skip sync for non-Calibre books + if (calibreId === null) { + return; + } + try { this.getCalibreService().updateRating(calibreId, rating); getLogger().info(`[BookService] Synced rating to Calibre (calibreId: ${calibreId}): ${rating ?? 'removed'}`); diff --git a/lib/services/duplicate-detection.service.ts b/lib/services/duplicate-detection.service.ts new file mode 100644 index 00000000..6ea2cd57 --- /dev/null +++ b/lib/services/duplicate-detection.service.ts @@ -0,0 +1,278 @@ +/** + * Duplicate Detection Service + * + * Detects potential duplicate books using Levenshtein distance algorithm. + * Used during manual book creation to warn users of possible duplicates. + * + * See: specs/003-non-calibre-books/spec.md (FR-009: Duplicate book detection) + */ + +import { getLogger } from "@/lib/logger"; +import { bookRepository } from "@/lib/repositories/book.repository"; + +const logger = getLogger().child({ module: "duplicate-detection" }); + +/** + * Levenshtein distance threshold for duplicate detection + * + * Similarity score above this threshold triggers duplicate warning. + * Range: 0-100, where 100 = identical strings + */ +const SIMILARITY_THRESHOLD = 85; + +/** + * Potential duplicate book + */ +export interface PotentialDuplicate { + bookId: number; + title: string; + authors: string[]; + source: string; + similarity: number; +} + +/** + * Duplicate detection result + */ +export interface DuplicateDetectionResult { + hasDuplicates: boolean; + duplicates: PotentialDuplicate[]; +} + +/** + * Calculate Levenshtein distance between two strings + * + * Returns the minimum number of single-character edits (insertions, + * deletions, or substitutions) required to change one string into the other. + * + * @param a - First string + * @param b - Second string + * @returns Levenshtein distance (lower = more similar) + */ +function levenshteinDistance(a: string, b: string): number { + const matrix: number[][] = []; + + // Initialize matrix + for (let i = 0; i <= b.length; i++) { + matrix[i] = [i]; + } + + for (let j = 0; j <= a.length; j++) { + matrix[0][j] = j; + } + + // Fill matrix + for (let i = 1; i <= b.length; i++) { + for (let j = 1; j <= a.length; j++) { + if (b.charAt(i - 1) === a.charAt(j - 1)) { + matrix[i][j] = matrix[i - 1][j - 1]; + } else { + matrix[i][j] = Math.min( + matrix[i - 1][j - 1] + 1, // substitution + matrix[i][j - 1] + 1, // insertion + matrix[i - 1][j] + 1 // deletion + ); + } + } + } + + return matrix[b.length][a.length]; +} + +/** + * Calculate similarity percentage between two strings + * + * Uses Levenshtein distance to compute similarity as a percentage. + * + * @param a - First string + * @param b - Second string + * @returns Similarity percentage (0-100, where 100 = identical) + */ +function calculateSimilarity(a: string, b: string): number { + const distance = levenshteinDistance(a.toLowerCase(), b.toLowerCase()); + const maxLength = Math.max(a.length, b.length); + + if (maxLength === 0) { + return 100; // Both strings empty + } + + const similarity = ((maxLength - distance) / maxLength) * 100; + return Math.round(similarity * 100) / 100; // Round to 2 decimal places +} + +/** + * Normalize book title for comparison + * + * Removes articles, punctuation, and extra whitespace for better matching. + * + * @param title - Book title + * @returns Normalized title + */ +function normalizeTitle(title: string): string { + return title + .toLowerCase() + .replace(/^(the|a|an)\s+/i, "") // Remove leading articles + .replace(/[^\w\s]/g, "") // Remove punctuation + .replace(/\s+/g, " ") // Normalize whitespace + .trim(); +} + +/** + * Normalize author name for comparison + * + * @param author - Author name + * @returns Normalized author name + */ +function normalizeAuthor(author: string): string { + return author + .toLowerCase() + .replace(/[^\w\s]/g, "") + .replace(/\s+/g, " ") + .trim(); +} + +/** + * Check if authors match + * + * Returns true if there's significant overlap in author names. + * + * @param authors1 - First author list + * @param authors2 - Second author list + * @returns True if authors match + */ +function authorsMatch(authors1: string[], authors2: string[]): boolean { + const normalized1 = authors1.map(normalizeAuthor); + const normalized2 = authors2.map(normalizeAuthor); + + // Check for exact matches + for (const author1 of normalized1) { + for (const author2 of normalized2) { + if (author1 === author2) { + return true; + } + + // Check for high similarity (>90%) + const similarity = calculateSimilarity(author1, author2); + if (similarity > 90) { + return true; + } + } + } + + return false; +} + +/** + * Detect potential duplicate books + * + * Searches for existing books with similar titles and matching authors. + * Uses Levenshtein distance with configurable similarity threshold. + * + * @param title - Book title to check + * @param authors - Book authors to check + * @param excludeBookId - Optional book ID to exclude (for updates) + * @returns Detection result with list of potential duplicates + */ +export async function detectDuplicates( + title: string, + authors: string[], + excludeBookId?: number +): Promise { + logger.debug({ title, authors, excludeBookId }, "Detecting duplicates"); + + const normalizedTitle = normalizeTitle(title); + const duplicates: PotentialDuplicate[] = []; + + // Fetch all books for comparison + // TODO: Optimize with fuzzy search or title index for large libraries (>10k books) + const allBooks = await bookRepository.findAll(); + + for (const book of allBooks) { + // Skip excluded book (for update scenarios) + if (excludeBookId && book.id === excludeBookId) { + continue; + } + + // Calculate title similarity + const bookNormalizedTitle = normalizeTitle(book.title); + const titleSimilarity = calculateSimilarity(normalizedTitle, bookNormalizedTitle); + + // Check if similarity exceeds threshold + if (titleSimilarity >= SIMILARITY_THRESHOLD) { + // Check if authors match + if (authorsMatch(authors, book.authors)) { + duplicates.push({ + bookId: book.id, + title: book.title, + authors: book.authors, + source: book.source, + similarity: titleSimilarity, + }); + + logger.info( + { + inputTitle: title, + existingTitle: book.title, + similarity: titleSimilarity, + bookId: book.id, + }, + "Potential duplicate detected" + ); + } + } + } + + // Sort duplicates by similarity (highest first) + duplicates.sort((a, b) => b.similarity - a.similarity); + + const result: DuplicateDetectionResult = { + hasDuplicates: duplicates.length > 0, + duplicates, + }; + + logger.debug( + { title, duplicateCount: duplicates.length }, + "Duplicate detection complete" + ); + + return result; +} + +/** + * Check if a specific book pair is likely a duplicate + * + * Convenience method for one-to-one comparison. + * + * @param title1 - First book title + * @param authors1 - First book authors + * @param title2 - Second book title + * @param authors2 - Second book authors + * @returns True if books are likely duplicates + */ +export function areBooksLikelyDuplicates( + title1: string, + authors1: string[], + title2: string, + authors2: string[] +): boolean { + const normalizedTitle1 = normalizeTitle(title1); + const normalizedTitle2 = normalizeTitle(title2); + + const titleSimilarity = calculateSimilarity(normalizedTitle1, normalizedTitle2); + + return titleSimilarity >= SIMILARITY_THRESHOLD && authorsMatch(authors1, authors2); +} + +/** + * Export internal functions for testing + * + * @internal + */ +export const _testing = { + levenshteinDistance, + calculateSimilarity, + normalizeTitle, + normalizeAuthor, + authorsMatch, + SIMILARITY_THRESHOLD, +}; diff --git a/lib/validation/manual-book.schema.ts b/lib/validation/manual-book.schema.ts new file mode 100644 index 00000000..6e2b1033 --- /dev/null +++ b/lib/validation/manual-book.schema.ts @@ -0,0 +1,106 @@ +/** + * Manual Book Validation Schemas + * + * Zod schemas for validating manual book input. + * + * See: specs/003-non-calibre-books/spec.md (FR-007: Manual book addition) + */ + +import { z } from "zod"; + +/** + * Manual book creation schema + * + * Required fields: + * - title: Non-empty string + * - authors: Array of at least one non-empty string + * + * Optional fields: + * - isbn: Valid ISBN-10 or ISBN-13 format + * - description: String + * - publisher: String + * - pubDate: ISO date string or Date object + * - totalPages: Integer between 1 and 10000 + * - series: String + * - seriesIndex: Positive number + * - tags: Array of strings + */ +export const manualBookSchema = z.object({ + // Required fields + title: z.string().min(1, "Title is required").max(500, "Title too long"), + authors: z + .array(z.string().min(1, "Author name cannot be empty")) + .min(1, "At least one author is required") + .max(20, "Too many authors"), + + // Optional fields + isbn: z + .string() + .regex( + /^(?:\d{9}[\dX]|\d{13})$/, + "Invalid ISBN format (must be ISBN-10 or ISBN-13)" + ) + .optional(), + + description: z.string().max(10000, "Description too long").optional(), + + publisher: z.string().max(200, "Publisher name too long").optional(), + + pubDate: z + .union([z.string().datetime(), z.date()]) + .transform((val) => (typeof val === "string" ? new Date(val) : val)) + .optional(), + + totalPages: z + .number() + .int("Page count must be an integer") + .min(1, "Page count must be at least 1") + .max(10000, "Page count too large") + .optional(), + + series: z.string().max(200, "Series name too long").optional(), + + seriesIndex: z + .number() + .positive("Series index must be positive") + .max(1000, "Series index too large") + .optional(), + + tags: z + .array(z.string().min(1, "Tag cannot be empty").max(50, "Tag too long")) + .max(50, "Too many tags") + .optional(), +}); + +/** + * Manual book update schema + * + * Same as creation schema but all fields optional (partial update support) + */ +export const manualBookUpdateSchema = manualBookSchema.partial(); + +/** + * Type inference from schema + */ +export type ManualBookInput = z.infer; +export type ManualBookUpdate = z.infer; + +/** + * Validation helper - returns parsed data or throws ZodError + */ +export function validateManualBookInput(data: unknown): ManualBookInput { + return manualBookSchema.parse(data); +} + +/** + * Safe validation helper - returns { success: true, data } or { success: false, errors } + */ +export function validateManualBookInputSafe(data: unknown): + | { success: true; data: ManualBookInput } + | { success: false; errors: z.ZodError } { + const result = manualBookSchema.safeParse(data); + if (result.success) { + return { success: true, data: result.data }; + } + return { success: false, errors: result.error }; +} From 9e9d4c5967e456a9ca5c170a0cee3d5fc598828e Mon Sep 17 00:00:00 2001 From: Mason Fox Date: Thu, 5 Feb 2026 18:47:59 -0500 Subject: [PATCH 09/89] feat(spec-003): Phase 3 Frontend - ManualBookForm and ProviderBadge components (T028-T033) - Add ManualBookForm component with full validation and duplicate detection - Add ProviderBadge component for visual source indicators - Implement real-time validation on blur - Add duplicate warning flow with 'Add Anyway' option - Integrate with BaseModal pattern and toast notifications - Support all book fields (required: title, authors; optional: ISBN, publisher, etc.) Co-authored-by: AI Assistant --- components/Books/ManualBookForm.tsx | 445 +++++++++++++++++++++++++ components/Providers/ProviderBadge.tsx | 93 ++++++ 2 files changed, 538 insertions(+) create mode 100644 components/Books/ManualBookForm.tsx create mode 100644 components/Providers/ProviderBadge.tsx diff --git a/components/Books/ManualBookForm.tsx b/components/Books/ManualBookForm.tsx new file mode 100644 index 00000000..890515e0 --- /dev/null +++ b/components/Books/ManualBookForm.tsx @@ -0,0 +1,445 @@ +"use client"; + +import { useState, useEffect } from "react"; +import BaseModal from "@/components/Modals/BaseModal"; +import { Button } from "@/components/Utilities/Button"; +import { toast } from "@/utils/toast"; +import { getLogger } from "@/lib/logger"; +import type { ManualBookInput } from "@/lib/validation/manual-book.schema"; +import type { PotentialDuplicate } from "@/lib/services/duplicate-detection.service"; + +const logger = getLogger().child({ component: "ManualBookForm" }); + +interface ManualBookFormProps { + isOpen: boolean; + onClose: () => void; + onSuccess: (bookId: number) => void; +} + +interface ValidationError { + path: (string | number)[]; + message: string; +} + +export default function ManualBookForm({ + isOpen, + onClose, + onSuccess, +}: ManualBookFormProps) { + // Form state + const [title, setTitle] = useState(""); + const [authors, setAuthors] = useState(""); + const [isbn, setIsbn] = useState(""); + const [publisher, setPublisher] = useState(""); + const [pubDate, setPubDate] = useState(""); + const [totalPages, setTotalPages] = useState(""); + const [series, setSeries] = useState(""); + const [seriesIndex, setSeriesIndex] = useState(""); + const [description, setDescription] = useState(""); + const [tags, setTags] = useState(""); + + // UI state + const [isSubmitting, setIsSubmitting] = useState(false); + const [validationErrors, setValidationErrors] = useState>({}); + const [duplicates, setDuplicates] = useState([]); + const [showDuplicateWarning, setShowDuplicateWarning] = useState(false); + + // Reset form when modal opens/closes + useEffect(() => { + if (!isOpen) { + setTitle(""); + setAuthors(""); + setIsbn(""); + setPublisher(""); + setPubDate(""); + setTotalPages(""); + setSeries(""); + setSeriesIndex(""); + setDescription(""); + setTags(""); + setValidationErrors({}); + setDuplicates([]); + setShowDuplicateWarning(false); + } + }, [isOpen]); + + // Real-time validation on blur (title and authors only) + const validateField = async (field: "title" | "authors") => { + if (field === "title" && !title.trim()) { + setValidationErrors((prev) => ({ ...prev, title: "Title is required" })); + return; + } + + if (field === "authors" && !authors.trim()) { + setValidationErrors((prev) => ({ ...prev, authors: "At least one author is required" })); + return; + } + + // Clear error if field is valid + setValidationErrors((prev) => { + const updated = { ...prev }; + delete updated[field]; + return updated; + }); + + // Check for duplicates if both title and authors are filled + if (title.trim() && authors.trim()) { + try { + const response = await fetch("/api/books/validate", { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ + title: title.trim(), + authors: authors.split(",").map((a) => a.trim()).filter((a) => a), + }), + }); + + if (response.ok) { + const data = await response.json(); + if (data.duplicates?.hasDuplicates) { + setDuplicates(data.duplicates.duplicates); + } else { + setDuplicates([]); + } + } + } catch (error) { + logger.error({ error }, "Failed to check for duplicates"); + } + } + }; + + const handleSubmit = async () => { + // Clear previous errors + setValidationErrors({}); + + // Parse authors + const authorList = authors + .split(",") + .map((a) => a.trim()) + .filter((a) => a); + + if (!title.trim()) { + setValidationErrors({ title: "Title is required" }); + return; + } + + if (authorList.length === 0) { + setValidationErrors({ authors: "At least one author is required" }); + return; + } + + // Show duplicate warning if duplicates exist and not already shown + if (duplicates.length > 0 && !showDuplicateWarning) { + setShowDuplicateWarning(true); + return; + } + + setIsSubmitting(true); + + try { + // Prepare payload + const payload: Partial = { + title: title.trim(), + authors: authorList, + }; + + // Add optional fields + if (isbn.trim()) payload.isbn = isbn.trim(); + if (publisher.trim()) payload.publisher = publisher.trim(); + if (pubDate.trim()) payload.pubDate = new Date(pubDate); + if (totalPages.trim()) payload.totalPages = parseInt(totalPages, 10); + if (series.trim()) payload.series = series.trim(); + if (seriesIndex.trim()) payload.seriesIndex = parseFloat(seriesIndex); + if (description.trim()) payload.description = description.trim(); + if (tags.trim()) { + payload.tags = tags.split(",").map((t) => t.trim()).filter((t) => t); + } + + // Submit to API + const response = await fetch("/api/books", { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify(payload), + }); + + if (!response.ok) { + const error = await response.json(); + + if (error.details) { + // Zod validation errors + const errors: Record = {}; + error.details.forEach((err: ValidationError) => { + const field = err.path[0] as string; + errors[field] = err.message; + }); + setValidationErrors(errors); + } else { + toast.error(error.error || "Failed to create book"); + } + return; + } + + const result = await response.json(); + logger.info({ bookId: result.book.id }, "Manual book created successfully"); + + toast.success(`"${result.book.title}" added to your library`); + onSuccess(result.book.id); + onClose(); + } catch (error) { + logger.error({ error }, "Failed to create manual book"); + toast.error("Failed to create book. Please try again."); + } finally { + setIsSubmitting(false); + } + }; + + const handleCancel = () => { + if (showDuplicateWarning) { + setShowDuplicateWarning(false); + } else { + onClose(); + } + }; + + return ( + + + + + } + > + {showDuplicateWarning ? ( +
+

+ The following {duplicates.length === 1 ? "book" : "books"} in your library {duplicates.length === 1 ? "appears" : "appear"} similar: +

+
+ {duplicates.map((dup) => ( +
+
+
+

+ {dup.title} +

+

+ by {dup.authors.join(", ")} +

+

+ Source: {dup.source} • {dup.similarity.toFixed(0)}% similar +

+
+
+
+ ))} +
+
+ ) : ( +
+ {/* Title - Required */} +
+ + setTitle(e.target.value)} + onBlur={() => validateField("title")} + className="w-full px-3 py-2 border border-gray-300 dark:border-gray-600 rounded-md bg-white dark:bg-gray-800 text-gray-900 dark:text-gray-100" + placeholder="Enter book title" + /> + {validationErrors.title && ( +

{validationErrors.title}

+ )} +
+ + {/* Authors - Required */} +
+ + setAuthors(e.target.value)} + onBlur={() => validateField("authors")} + className="w-full px-3 py-2 border border-gray-300 dark:border-gray-600 rounded-md bg-white dark:bg-gray-800 text-gray-900 dark:text-gray-100" + placeholder="e.g., Jane Doe, John Smith (comma-separated)" + /> + {validationErrors.authors && ( +

{validationErrors.authors}

+ )} +
+ + {/* Optional fields in grid */} +
+ {/* ISBN */} +
+ + setIsbn(e.target.value)} + className="w-full px-3 py-2 border border-gray-300 dark:border-gray-600 rounded-md bg-white dark:bg-gray-800 text-gray-900 dark:text-gray-100" + placeholder="ISBN-10 or ISBN-13" + /> + {validationErrors.isbn && ( +

{validationErrors.isbn}

+ )} +
+ + {/* Total Pages */} +
+ + setTotalPages(e.target.value)} + className="w-full px-3 py-2 border border-gray-300 dark:border-gray-600 rounded-md bg-white dark:bg-gray-800 text-gray-900 dark:text-gray-100" + placeholder="e.g., 350" + /> + {validationErrors.totalPages && ( +

{validationErrors.totalPages}

+ )} +
+
+ +
+ {/* Publisher */} +
+ + setPublisher(e.target.value)} + className="w-full px-3 py-2 border border-gray-300 dark:border-gray-600 rounded-md bg-white dark:bg-gray-800 text-gray-900 dark:text-gray-100" + placeholder="e.g., Penguin Books" + /> +
+ + {/* Publication Date */} +
+ + setPubDate(e.target.value)} + className="w-full px-3 py-2 border border-gray-300 dark:border-gray-600 rounded-md bg-white dark:bg-gray-800 text-gray-900 dark:text-gray-100" + /> +
+
+ +
+ {/* Series */} +
+ + setSeries(e.target.value)} + className="w-full px-3 py-2 border border-gray-300 dark:border-gray-600 rounded-md bg-white dark:bg-gray-800 text-gray-900 dark:text-gray-100" + placeholder="e.g., Harry Potter" + /> +
+ + {/* Series Index */} +
+ + setSeriesIndex(e.target.value)} + className="w-full px-3 py-2 border border-gray-300 dark:border-gray-600 rounded-md bg-white dark:bg-gray-800 text-gray-900 dark:text-gray-100" + placeholder="e.g., 1" + /> +
+
+ + {/* Tags */} +
+ + setTags(e.target.value)} + className="w-full px-3 py-2 border border-gray-300 dark:border-gray-600 rounded-md bg-white dark:bg-gray-800 text-gray-900 dark:text-gray-100" + placeholder="e.g., fiction, fantasy (comma-separated)" + /> +
+ + {/* Description */} +
+ +