From 64179aeaeab1fcf72c5408f294756b84b6390cf9 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 1 Mar 2026 20:23:05 +0000 Subject: [PATCH] Add comprehensive project analysis and enhancement proposals Thorough analysis of the full codebase (backend, frontend, infrastructure) with 30+ actionable enhancement ideas organized by category and priority, including a phased roadmap for implementation. https://claude.ai/code/session_01UqkzRrrmJyChpoz42gmeKE --- ENHANCEMENTS.md | 609 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 609 insertions(+) create mode 100644 ENHANCEMENTS.md diff --git a/ENHANCEMENTS.md b/ENHANCEMENTS.md new file mode 100644 index 0000000..76c4dbc --- /dev/null +++ b/ENHANCEMENTS.md @@ -0,0 +1,609 @@ +# RAW-Manager: Project Analysis & Enhancement Proposals + +> Comprehensive analysis of the codebase with actionable improvement ideas, +> organized by category and priority. + +--- + +## Table of Contents + +1. [Backend Enhancements](#1-backend-enhancements) +2. [Frontend & UX Enhancements](#2-frontend--ux-enhancements) +3. [New Feature Ideas](#3-new-feature-ideas) +4. [Performance Optimizations](#4-performance-optimizations) +5. [Reliability & Error Handling](#5-reliability--error-handling) +6. [Infrastructure & DevOps](#6-infrastructure--devops) +7. [Testing & Quality](#7-testing--quality) +8. [Accessibility](#8-accessibility) +9. [Quick Wins](#9-quick-wins) + +--- + +## 1. Backend Enhancements + +### 1.1 Streaming ZIP Downloads (High Priority) + +**Current state:** `POST /api/download` buffers the entire ZIP in memory via `io.BytesIO()`. For 200 large RAW files, this can consume gigabytes of RAM. + +**Proposal:** Use `StreamingResponse` with a generator that yields ZIP chunks as they're written. FastAPI supports this natively: + +```python +from fastapi.responses import StreamingResponse + +def zip_generator(photo_paths): + # Use zipfile with a pipe or tempfile approach + ... + +return StreamingResponse(zip_generator(paths), media_type="application/zip") +``` + +**Impact:** Eliminates OOM risk for large batch downloads. Allows progress tracking on the client side. + +--- + +### 1.2 HTTP Cache Headers for Thumbnails & Originals (High Priority) + +**Current state:** `/api/files/thumb/{id}` and `/api/files/original/{id}` return `FileResponse` with no caching directives. Every gallery scroll re-fetches thumbnails from the server. + +**Proposal:** Add `Cache-Control`, `ETag` (based on `file_hash`), and `Last-Modified` headers. Thumbnails are immutable (content-addressed by SHA256), so they can use aggressive caching: + +``` +Cache-Control: public, max-age=31536000, immutable +ETag: "sha256-" +``` + +**Impact:** Dramatically reduces bandwidth and server load for repeat gallery browsing. Browser caches thumbnails indefinitely. + +--- + +### 1.3 Database Schema Versioning (Medium Priority) + +**Current state:** Migrations are handled by a single try/except block that attempts `SELECT dev_status` and runs `ALTER TABLE` on failure. This doesn't scale as the schema evolves. + +**Proposal:** Add a `schema_version` table and a simple migration runner: + +```sql +CREATE TABLE IF NOT EXISTS schema_version ( + version INTEGER PRIMARY KEY, + applied_at TEXT DEFAULT (datetime('now')) +); +``` + +Each migration is a numbered function. On startup, run any migrations with version > current. + +**Impact:** Safe, repeatable schema evolution. Makes it trivial to add new columns or indexes in future releases. + +--- + +### 1.4 Consolidated Stats Query (Low Priority) + +**Current state:** `get_stats()` in `database.py` runs 4 separate queries to count total, rated, kept, and discarded photos plus an AVG query. + +**Proposal:** Combine into a single aggregation: + +```sql +SELECT + COUNT(*) as total, + SUM(CASE WHEN rating > 0 THEN 1 ELSE 0 END) as rated, + SUM(CASE WHEN status = 'keep' THEN 1 ELSE 0 END) as kept, + SUM(CASE WHEN status = 'discard' THEN 1 ELSE 0 END) as discarded, + ROUND(AVG(CASE WHEN rating > 0 THEN rating END), 1) as avg_rating +FROM photos; +``` + +**Impact:** 5x fewer DB round-trips for the stats endpoint. Noticeable improvement on large libraries. + +--- + +### 1.5 Add `.cr3` and `.heif` RAW Support (Medium Priority) + +**Current state:** `SUPPORTED_RAW_EXTENSIONS` includes 7 formats but is missing `.cr3` (Canon EOS R series, widely adopted since 2018) and `.heif`/`.hif` (some newer cameras). + +**Proposal:** Add `.cr3` to `raw_processing.py`. Verify `rawpy`/LibRaw version supports it (LibRaw 0.20+ does). Consider `.heif` as experimental. + +**Impact:** Supports photographers with modern Canon mirrorless cameras (EOS R5, R6, R7, R8, etc.). + +--- + +### 1.6 Batch/Bulk API Endpoints (Medium Priority) + +**Current state:** Rating or changing status on multiple photos requires N individual `PATCH /api/photos/{id}` calls. + +**Proposal:** Add `PATCH /api/photos/batch` accepting `{"ids": [...], "updates": {"rating": 3}}`. Execute in a single transaction. + +**Impact:** Selection + batch rate/status in the UI becomes a single API call instead of potentially hundreds. Reduces latency and eliminates partial-failure states. + +--- + +## 2. Frontend & UX Enhancements + +### 2.1 Lightbox Zoom & Pan (High Priority) + +**Current state:** The Lightbox shows full-screen images but provides no zoom or pan controls. Photographers can't inspect focus or fine detail. + +**Proposal:** +- Mouse wheel / pinch-to-zoom on the image +- Click-and-drag to pan when zoomed +- Double-click to toggle 100% zoom at click point +- Display zoom level indicator + +**Impact:** Essential for photo culling workflows. Photographers need to check sharpness at pixel level. + +--- + +### 2.2 Photo Compare Mode (Medium Priority) + +**Current state:** No way to compare two similar shots side-by-side. + +**Proposal:** When 2 photos are selected, offer a "Compare" button that opens a split-screen lightbox: +- Side-by-side view (default) +- Synchronized zoom & pan (move one, both move) +- Swipe divider overlay mode +- Quick keep/discard on each side + +**Impact:** Core photography workflow for choosing the best shot from a burst or similar compositions. + +--- + +### 2.3 Drag-and-Drop Upload (Medium Priority) + +**Current state:** Photos must be placed in `IMPORT_DIR` via the filesystem. No web-based upload. + +**Proposal:** Add a drag-and-drop zone (or click-to-upload) in the UI that uploads RAW files via a new `POST /api/upload` endpoint. The endpoint saves files to `IMPORT_DIR` and the ingestion service picks them up. + +**Impact:** Enables remote photo management without filesystem access. Important for Docker deployments where users may not have direct volume access. + +--- + +### 2.4 Toast Notifications (Low Priority) + +**Current state:** Actions like rating, status changes, and batch operations have no feedback beyond the UI state change. Errors are logged to console only. + +**Proposal:** Add a lightweight toast/snackbar system: +- Success: "Rated 5 photos as 4 stars" +- Error: "Failed to update photo — server unreachable" +- Info: "Database rebuild complete: 1,204 photos processed" + +**Impact:** Better user confidence that actions succeeded. Surfaces errors that currently go unnoticed. + +--- + +### 2.5 Smart Selection Helpers (Low Priority) + +**Current state:** Selection is manual — click individual cards or use Shift+click. + +**Proposal:** +- "Select all visible" button +- "Select all matching filters" button +- "Select all rated >= N" shortcut +- "Invert selection" button +- Shift+click for range selection + +**Impact:** Faster batch workflows for large libraries. + +--- + +## 3. New Feature Ideas + +### 3.1 Tags / Labels System (High Priority) + +**Current state:** Photos can only be categorized by rating (0-5), status (pending/keep/discard), and dev_status. No free-form organization. + +**Proposal:** +- New `tags` table: `id`, `name`, `color` +- Junction table: `photo_tags` (`photo_id`, `tag_id`) +- API: `GET/POST/DELETE /api/tags`, `POST /api/photos/{id}/tags` +- UI: Tag chips on photo cards, tag filter in toolbar, tag management in settings +- Keyboard shortcut: `T` to open tag picker in lightbox + +**Impact:** Enables project-based workflows ("wedding", "landscape", "portfolio"), subject categorization, and custom organization beyond the linear rating scale. + +--- + +### 3.2 Smart Albums / Saved Filters (Medium Priority) + +**Current state:** Filters are ephemeral — they reset when you navigate away. + +**Proposal:** +- Save current filter combination as a named "smart album" +- New `albums` table: `id`, `name`, `filters_json`, `created_at` +- Sidebar or dropdown to quickly switch between saved filters +- Examples: "Best of 2025", "Unculled from last week", "Portfolio candidates" + +**Impact:** Saves time for recurring workflows. Photographers often revisit the same filter sets. + +--- + +### 3.3 EXIF-Based Auto-Grouping (Medium Priority) + +**Current state:** Photos are grouped only by month in the gallery. + +**Proposal:** Offer additional grouping modes: +- **By camera** — useful for multi-camera shoots +- **By lens** — see which glass produced which shots +- **By date (day-level)** — finer than month +- **By burst/sequence** — group photos taken within seconds of each other + +**Impact:** Helps photographers analyze their shooting patterns and cull more efficiently. + +--- + +### 3.4 Color Label System (Low Priority) + +**Current state:** Only star ratings and text statuses exist. + +**Proposal:** Add a color label field (red, yellow, green, blue, purple) similar to Lightroom/Capture One. These are commonly used in photography workflows: +- Red = reject on second pass +- Yellow = maybe +- Green = selected +- Blue = edited +- Purple = client pick + +**Impact:** Familiar workflow for photographers migrating from other tools. + +--- + +### 3.5 Duplicate Detection UI (Low Priority) + +**Current state:** Deduplication happens at ingest time via SHA256. But visually similar (not identical) photos are not detected. + +**Proposal:** Add a "Find Similar" feature using perceptual hashing (pHash/dHash): +- Compute perceptual hash during thumbnail generation +- New endpoint: `GET /api/photos/{id}/similar` +- UI: "Similar photos" section in lightbox sidebar +- Configurable similarity threshold + +**Impact:** Helps identify near-duplicates from burst shooting or slight recompositions. + +--- + +## 4. Performance Optimizations + +### 4.1 Frontend Request Caching (High Priority) + +**Current state:** `usePhotos.js` makes a fresh API call every time filters change, even when switching back to a previously-viewed filter combination. No caching layer exists. + +**Proposal:** Implement a simple LRU cache keyed by the serialized filter+pagination state. Cache the last 10-20 result sets. Invalidate on mutations (rate, delete, status change). + +Alternatively, adopt `@tanstack/react-query` (TanStack Query) which provides: +- Automatic request deduplication +- Background refetching +- Cache invalidation +- Optimistic updates + +**Impact:** Switching between filters feels instant. Reduces API load significantly. + +--- + +### 4.2 Indexed Date Queries (Medium Priority) + +**Current state:** Date filtering uses `REPLACE(date_taken, ':', '-')` in WHERE clauses, which runs a string transformation on every row and prevents index usage. + +**Proposal:** Store dates in a consistent ISO-8601 format (already done for most entries) and add a computed/stored column or ensure the `date_taken` index is usable without transformation. Alternatively, add a `date_taken_normalized` column populated at insert time. + +**Impact:** Date range queries and timeline grouping become index-scannable instead of full table scans. + +--- + +### 4.3 Thumbnail Preloading in Lightbox (Medium Priority) + +**Current state:** When navigating the lightbox with arrow keys, the next image loads only after the key press. + +**Proposal:** Preload the previous and next 2-3 thumbnails using `` or `new Image()` in JavaScript. The URLs are known from the photo array. + +**Impact:** Near-instant navigation in lightbox. Critical for fast culling workflows. + +--- + +### 4.4 Virtual Scrolling for Large Libraries (Medium Priority) + +**Current state:** `react-virtuoso` is installed as a dependency but not used. The gallery renders all loaded cards in the DOM (potentially thousands). + +**Proposal:** Either: +1. Implement virtual scrolling using the already-installed `react-virtuoso` for the gallery grid +2. Or remove the unused dependency and keep the current IntersectionObserver approach (which works well enough with the existing VirtualCard lazy-render pattern) + +**Impact:** Option 1 reduces DOM nodes from thousands to ~50, improving scroll performance. Option 2 reduces bundle size. + +--- + +### 4.5 Exiftool Stay-Open Mode (Low Priority) + +**Current state:** A new `exiftool` process is spawned for every file during ingestion. + +**Proposal:** Use exiftool's `-stay_open` flag to keep a single process running and pipe file paths to it. This eliminates per-file process creation overhead. + +**Impact:** ~10x faster metadata extraction during bulk imports or rebuilds. Most noticeable when importing 100+ files. + +--- + +## 5. Reliability & Error Handling + +### 5.1 Filesystem + Database Transaction Safety (High Priority) + +**Current state:** Operations that modify both the database and filesystem (delete photo, darkroom copy/remove) have no transactional guarantees. If the DB update succeeds but file deletion fails, the system enters an inconsistent state. + +**Proposal:** Implement a "write-ahead" pattern: +1. Write intention to a journal/log +2. Perform filesystem operation +3. Update database +4. Clear journal entry + +On startup, replay any incomplete journal entries. + +Alternatively, for simpler operations: perform the filesystem operation first, then update the DB. If the DB update fails, the filesystem operation can be retried. + +**Impact:** Eliminates orphaned files and ghost database entries after crashes or errors. + +--- + +### 5.2 Ingestion Retry with Backoff (Medium Priority) + +**Current state:** If metadata extraction or thumbnail generation fails, the file is skipped permanently with a log message. No retry. + +**Proposal:** Implement a retry queue with exponential backoff: +- Transient failures (disk full, exiftool timeout): retry 3 times with 5s/30s/120s delays +- Permanent failures (corrupted RAW): mark as failed, surface in UI +- New endpoint: `GET /api/ingestion/failures` to list failed imports + +**Impact:** Recovers from transient errors automatically. Failed files are visible instead of silently lost. + +--- + +### 5.3 Thread-Safe Rebuild State (Medium Priority) + +**Current state:** `_rebuild_state` dict in `main.py` has inconsistent locking. Some reads happen outside the lock, creating race conditions if two rebuild requests arrive simultaneously. + +**Proposal:** Either: +1. Use a proper `dataclass` with a `threading.Lock` that's always acquired for any read/write +2. Or switch to an `asyncio`-based approach using FastAPI's async capabilities + +**Impact:** Prevents corrupt state during concurrent rebuild/status-check requests. + +--- + +### 5.4 Graceful Shutdown for Ingestion (Low Priority) + +**Current state:** The ingestion service catches `KeyboardInterrupt` and calls `observer.stop()`, but files currently being processed may be left in an incomplete state. + +**Proposal:** Track in-progress files and either: +- Complete processing before shutting down (with a timeout) +- Or move partially processed files back to a retry queue + +**Impact:** No orphaned files or partial database entries on service restart. + +--- + +## 6. Infrastructure & DevOps + +### 6.1 Add CI/CD Pipeline (High Priority) + +**Current state:** No CI/CD exists. No automated testing, linting, or builds. + +**Proposal:** Add GitHub Actions workflow: + +```yaml +# .github/workflows/ci.yml +- Backend: pytest, ruff (linting), mypy (type checking) +- Frontend: npm test (vitest), eslint +- Docker: build & smoke test +- On PR: run all checks +- On main merge: build & push Docker image +``` + +**Impact:** Catches regressions before merge. Builds confidence in contributions. + +--- + +### 6.2 Database Backup Strategy (High Priority) + +**Current state:** SQLite database lives on a Docker volume with no backup mechanism. A volume deletion or corruption loses all metadata, ratings, and curation work. + +**Proposal:** +- Add a `POST /api/backup` endpoint that creates a timestamped `.sqlite3` backup +- Schedule automatic daily backups via cron in the container +- Expose backup files via `GET /api/backups` for download +- SQLite's online backup API (`sqlite3.backup()`) allows safe hot backups + +**Impact:** Protects potentially months of curation work. Critical for any production use. + +--- + +### 6.3 Environment Variable Validation at Startup (Medium Priority) + +**Current state:** Environment variables are read with `os.getenv()` defaults scattered across files. Invalid values (e.g., `THUMB_SIZE=abc`) fail silently or at runtime. + +**Proposal:** Add a startup validation step that: +- Checks all required directories exist or can be created +- Validates numeric variables are in expected ranges +- Verifies `exiftool` is installed and accessible +- Logs the full configuration on startup + +**Impact:** Fail-fast on misconfiguration instead of mysterious runtime errors. + +--- + +### 6.4 Separate Import Volume in Docker Compose (Medium Priority) + +**Current state:** Both services share a single `photo_data` volume. The import directory is inside this volume, meaning external tools can't easily drop files without Docker volume access. + +**Proposal:** Add a separate bind mount for the import directory: + +```yaml +volumes: + - photo_data:/app/data + - ./import:/app/data/import # Accessible from host filesystem +``` + +**Impact:** Users can drag-and-drop files from their host OS directly into the import folder. + +--- + +### 6.5 Multi-Architecture Docker Build (Low Priority) + +**Current state:** Dockerfile builds for the host platform only. + +**Proposal:** Add `docker buildx` support for `linux/amd64` and `linux/arm64`. This enables running on Raspberry Pi, Apple Silicon Macs (via Docker Desktop), and ARM-based NAS devices. + +**Impact:** Expands deployment options for home network use cases (the project's target audience). + +--- + +## 7. Testing & Quality + +### 7.1 Backend Unit Tests (High Priority) + +**Current state:** `backend/tests/` exists with only `__init__.py`. Zero test coverage. + +**Proposal:** Priority test targets: + +1. **`database.py`** — Most testable, pure functions with `db_path` parameter + - `init_db()` creates schema correctly + - `insert_photo()` handles duplicates + - `get_photos()` with various filter combinations + - `update_photo()` respects allow-list + - `get_timeline()` groups correctly + +2. **`utils/file_utils.py`** — Pure utility functions + - `compute_hash()` produces correct SHA256 + - `safe_copy()` verifies integrity + - `organize_path()` handles filename conflicts + +3. **`utils/exif_utils.py`** — Mock `subprocess.run` for exiftool + - Metadata normalization across camera brands + - Format functions for shutter speed, focal length + +4. **`main.py`** — FastAPI TestClient integration tests + - Photo CRUD lifecycle + - Filter combinations + - Error responses (404, 400, 409) + +**Impact:** Enables confident refactoring and catches regressions. + +--- + +### 7.2 Frontend Testing Setup (Medium Priority) + +**Current state:** No testing framework configured. + +**Proposal:** Add Vitest + React Testing Library: +- Test `usePhotos` hook with mocked `fetch` +- Test `StarRating` interactions +- Test `useKeyboardShortcuts` key bindings +- Test filter state management in `Toolbar` + +**Impact:** Frontend stability as features are added. + +--- + +### 7.3 Add Linting (Medium Priority) + +**Current state:** No linting configured for either backend or frontend. + +**Proposal:** +- Backend: `ruff` (fast Python linter + formatter) +- Frontend: `eslint` with React plugin +- Add pre-commit hooks or CI checks + +**Impact:** Consistent code style, catches common bugs automatically. + +--- + +## 8. Accessibility + +### 8.1 Keyboard Navigation (High Priority) + +**Current issues identified:** +- `PhotoCard` has `role="button"` but doesn't respond to Enter/Space keys +- `TimelineScrubber` has no keyboard support at all (arrow keys, Enter) +- `Lightbox` doesn't trap or manage focus on open/close +- Star ratings lack `aria-label` per individual star button + +**Proposal:** +- Add `onKeyDown` handlers to PhotoCard for Enter/Space activation +- Add arrow key navigation to TimelineScrubber +- Implement focus trap in Lightbox (focus first interactive element on open, restore focus on close) +- Add `aria-label="Rate {n} of 5 stars"` to each star button + +**Impact:** Makes the app usable for keyboard-only users and screen reader users. + +--- + +### 8.2 ARIA Labels & Roles (Medium Priority) + +**Current issues:** +- Settings modal lacks `role="dialog"` and `aria-modal="true"` +- Progress bar in rebuild dialog has no `aria-valuenow`/`aria-valuemin`/`aria-valuemax` +- Filter dropdowns in Toolbar lack associated `