v0.7.x: Migration Map, AI Discovery Dismiss, Debug Endpoints

.changelog/v0.6.x.md → .changelog/v0.6.33.md

@@ -1,4 +1,4 @@
-# Release v0.6.x - FamilySearch Normalization & Architecture Refinement
+# Release v0.6.33 - FamilySearch Normalization & Architecture Refinement
 
 Released: YYYY-MM-DD
 
@@ -8,7 +8,7 @@ This release normalizes FamilySearch as a downstream provider (equal to Ancestry
 
 ## New Features
 
-### Ancestry Update Automation Page (v0.6.x)
+### Ancestry Update Automation Page (v0.6.33)
 - New database-scoped page at `/db/:dbId/ancestry-update` for bulk Ancestry synchronization
 - BFS traversal of direct-ancestor sparse tree from selected root person
 - Configurable generation depth (1-4 or full tree)
@@ -18,15 +18,15 @@ This release normalizes FamilySearch as a downstream provider (equal to Ancestry
 - Adds "Ancestry Update" link to database sidebar
 - API endpoints: GET `/api/ancestry-update/status`, GET `/:dbId/events` (SSE), POST `/:dbId/cancel`, GET `/:dbId/validate/:personId`
 
-### Ancestry Free Hints Automation (v0.6.x)
+### Ancestry Free Hints Automation (v0.6.33)
 - New "Hints" button on Ancestry row in Provider Data table to process free record hints
 - Playwright automation navigates to hints page, reviews each hint, accepts and saves to tree
 - Supports checkbox selection for adding related people from hints
 - Includes consecutive failure safeguard (stops after 3 failures) and remaining hints re-check
 - SSE-based progress events for real-time UI updates
 - API endpoints: POST/GET `/api/ancestry-hints/:dbId/:personId`, cancel, status
 
-### Ancestry Vital Info Upload (v0.6.x)
+### Ancestry Vital Info Upload (v0.6.33)
 - Extended Ancestry upload to support birth date, birth place, death date, death place, and name (not just photos)
 - Upload dialog now shows field-by-field comparison between local SparseTree data and cached Ancestry data
 - Browser automation uses Ancestry's Quick Edit sidebar to update person facts
@@ -39,69 +39,69 @@ This release normalizes FamilySearch as a downstream provider (equal to Ancestry
 - Added `linkFamilySearch()` and `getFamilySearchPhotoPath()` to augmentation service
 - ID resolution uses alphabetical order (no FamilySearch priority)
 
-### Architecture: Source of Truth Principle (v0.6.x)
+### Architecture: Source of Truth Principle (v0.6.33)
 - SparseTree SQLite is now the canonical source of truth
 - Provider cache (`data/provider-cache/`) is for comparison only, not a fallback data source
 - All data reads come from SQLite; provider cache shows what providers have vs what SparseTree has
 
 ## Bug Fixes
 
-### SQLite Vital Event Row ID Preservation (v0.6.x)
+### SQLite Vital Event Row ID Preservation (v0.6.33)
 - Changed vital_event INSERT from `INSERT OR REPLACE` to `ON CONFLICT DO UPDATE`
 - Preserves row IDs which are critical for local_override references
 - INSERT OR REPLACE was deleting and re-inserting rows, orphaning override references
 
-### Improved Vital Event Date Parsing (v0.6.x)
+### Improved Vital Event Date Parsing (v0.6.33)
 - Now uses actual `birth.date` and `death.date` objects instead of parsing lifespan string
 - Captures full dates (e.g., "1979-07-31") rather than just years from lifespan ("1979-2020")
 - Added `parseYearFromDate()` function to handle various date formats including BC notation
 
-### Ancestry Update Log Memory Fix (v0.6.x)
+### Ancestry Update Log Memory Fix (v0.6.33)
 - Limited execution log entries to 500 to prevent browser memory issues on large trees
 - Shows notice when older entries are truncated
 
-### FamilySearch Upload Improvements (v0.6.x)
+### FamilySearch Upload Improvements (v0.6.33)
 - Auto-refresh FamilySearch cache after successful upload so UI shows updated data
 - Improved photo comparison to detect already-uploaded photos via symlink matching
 - Fixed Google login button selector for updated FamilySearch login page
 - Improved vitals page editing: handles conclusion dialogs and edit buttons
 - Added autocomplete handling for date/place fields (clicks first suggestion)
 
-### Scraper Auto-Login (v0.6.x)
+### Scraper Auto-Login (v0.6.33)
 - Scraper now attempts auto-login via Google OAuth when landing on FamilySearch login page
 - Added `handleLoginIfNeeded()` function to detect login pages and complete OAuth flow
 - Navigates directly to target URL after successful login
 
-### Provider Comparison Not Using Local Overrides (v0.6.x)
+### Provider Comparison Not Using Local Overrides (v0.6.33)
 - Fixed comparison service using original database values instead of user-applied overrides
 - Added `applyLocalOverrides()` to fetch and apply vital_event overrides before comparison
 - FamilySearch/Ancestry now correctly show "match" when user has applied their values
 
-### Date Format Differences Showing as Different (v0.6.x)
+### Date Format Differences Showing as Different (v0.6.33)
 - Added date normalization to expand month abbreviations (e.g., "AUG" → "august")
 - "29 AUG 1933" now correctly matches "29 August 1933"
 - Reduces false positives in provider comparison diff counts
 
-### Unicode Escape in View Mode Selector (v0.6.x)
+### Unicode Escape in View Mode Selector (v0.6.33)
 - Fixed checkmark showing as literal `\u2713` text instead of ✓ in tree view mode dropdown
 - JSX text content requires actual Unicode characters, not escape sequences
 
-### Fan Chart Ancestor Count (v0.6.x)
+### Fan Chart Ancestor Count (v0.6.33)
 - Fixed footer showing incorrect count like "109 of 62 ancestors shown"
 - Now correctly counts only ancestors within the visible generation range
 
-### Stale Browser Connection Detection (v0.6.x)
+### Stale Browser Connection Detection (v0.6.33)
 - Added `verifyAndReconnect()` to browser service that checks both Playwright state and actual browser process
 - Augmentation service now detects and recovers from stale connections before scraping
 - Fixes silent failures when browser was closed but connection state was still "connected"
 
-### Auto-Login on Sync (v0.6.x)
+### Auto-Login on Sync (v0.6.33)
 - FamilySearch sync now attempts automatic login when auth token is missing
 - Uses `providerService.ensureAuthenticated()` to try Google SSO or stored credentials
 - Eliminates need for manual browser login before sync operations
 - Applies to both `refreshPerson()` and `fetchPersonDisplayName()` methods
 
-### FamilySearch Scraper Not Extracting Data (v0.6.x)
+### FamilySearch Scraper Not Extracting Data (v0.6.33)
 - Updated scraper selectors to match current FamilySearch DOM structure
 - Name now extracted using `[data-testid="fullName"]`
 - Birth/death dates now extracted from `[data-testid="conclusionDisplay:BIRTH"]` and `[data-testid="conclusionDisplay:DEATH"]` containers
@@ -111,15 +111,15 @@ This release normalizes FamilySearch as a downstream provider (equal to Ancestry
 - Added detailed debug logging for photo extraction troubleshooting
 - Improved UI feedback: loading toast during scrape, success toast shows extracted fields
 
-### "Use" Button Not Updating SparseTree Row (v0.6.x)
+### "Use" Button Not Updating SparseTree Row (v0.6.33)
 - Fixed clicking "Use" on provider field values not updating the SparseTree row display
 - Root causes fixed:
   - Client: `onFieldChanged` callback now refetches person data after applying override
   - Server: `getPerson()` now applies local overrides to returned person data
 - Added `applyLocalOverridesToPerson()` to database service for consistent override application
 - Birth/death date/place values from providers now properly display after clicking "Use"
 
-### "Set as Primary" Photo Not Updating (v0.6.x)
+### "Set as Primary" Photo Not Updating (v0.6.33)
 - Fixed photo not updating after clicking "Set as Primary" on provider photos
 - Root causes fixed:
   - `getPhotoPath()` now checks for primary photo (`{personId}.jpg`) before provider-suffixed photos
@@ -129,27 +129,27 @@ This release normalizes FamilySearch as a downstream provider (equal to Ancestry
 - Added client API methods: `getFsPhotoUrl()`, `hasFsPhotoProvider()`
 - Added `photoVersion` state for cache-busting photo URLs after changes
 
-### HMR Context Resilience (v0.6.x)
+### HMR Context Resilience (v0.6.33)
 - Fixed "useSidebar must be used within a SidebarProvider" error during HMR transitions
 - `useSidebar()` and `useTheme()` hooks now return safe defaults during dev hot reloads
 - Prevents crashes when React context identity changes during development
 
 ## Improvements
 
-### FamilySearch Upload Performance (v0.6.x)
+### FamilySearch Upload Performance (v0.6.33)
 - Eliminated redundant post-login delays in FamilySearch upload automation
 - After Google OAuth redirect, now checks if already on target page before re-navigating
 - Replaced fixed `waitForTimeout` delays with element-based `waitForSelector` for responsiveness
 - Added PM2 timestamps to server logs for better debugging (`time: true` in ecosystem config)
 
-### Page-Level Padding Control (v0.6.x)
+### Page-Level Padding Control (v0.6.33)
 - Moved `p-6` padding from Layout to individual pages for flexibility
 - Full-height pages (tree views, PersonDetail, SparseTreePage) control their own layout
 - Standard content pages receive consistent padding via their root element
 - Fixed missing padding on SparseTreePage and PersonDetail
 - Removed double padding from AIProviders page (external component handles its own padding)
 
-### Tree Visualization Parity with Ancestry.com (v0.6.x)
+### Tree Visualization Parity with Ancestry.com (v0.6.33)
 - Added 5 tree view modes matching Ancestry.com functionality
 - **Fan Chart** (default): Radial pedigree with lineage-colored wedges (paternal=blue/teal, maternal=red/coral)
 - **Horizontal Pedigree**: Root left, ancestors right with expandable nodes
@@ -160,7 +160,7 @@ This release normalizes FamilySearch as a downstream provider (equal to Ancestry
 - New utilities: lineageColors, treeLayout, arcGenerator
 - Fixed classic view bugs: dynamic centering, ResizeObserver instead of setTimeout
 
-### Vertical Family View Improvements (v0.6.x)
+### Vertical Family View Improvements (v0.6.33)
 - Refactored layout to use multi-pass collision resolution algorithm
 - Parents properly avoid overlapping when multiple branches are expanded
 - Added horizontal coupling bars between parent pairs (matching Ancestry.com style)
@@ -170,13 +170,13 @@ This release normalizes FamilySearch as a downstream provider (equal to Ancestry
 - Fixed expand/collapse button positioning to overlap card tops consistently
 - Moved spouse connection line from middle of card to name/date area
 
-### Avatar Placeholders (v0.6.x)
+### Avatar Placeholders (v0.6.33)
 - Replaced emoji-based avatar placeholders with proper SVG components throughout tree views
 - New `AvatarPlaceholder` component renders gender-aware silhouettes (male/female/unknown profiles)
 - Updated PersonCard, FocusNavigatorView, GenerationalColumnsView, PedigreeChartView, and SparseTreePage
 - Softened gender color palette for better visual harmony (less saturated blue/pink)
 
-### Legacy Code Removal (v0.6.x)
+### Legacy Code Removal (v0.6.33)
 - Removed all legacy fallback code from services (no more dual-path lookups)
 - Services updated: `familysearch-refresh`, `multi-platform-comparison`, `augmentation`, `scraper`, `person.routes`
 - Architecture policy: use migration scripts instead of maintaining legacy paths
@@ -213,4 +213,4 @@ This release normalizes FamilySearch as a downstream provider (equal to Ancestry
 
 ## Full Changelog
 
-**Full Diff**: https://github.com/atomantic/SparseTree/compare/v0.5.4...v0.6.x
+**Full Diff**: https://github.com/atomantic/SparseTree/compare/v0.5.4...v0.6.33

.changelog/v0.7.x.md

@@ -0,0 +1,46 @@
+# Release v0.7.x - Migration Map, AI Discovery Improvements
+
+Released: Unreleased
+
+## Overview
+
+This release improves the AI Discovery feature with dismiss functionality, debugging capabilities, and better CLI integration.
+
+## New Features
+
+### AI Discovery Dismiss Functionality (v0.7.0)
+- Dismiss AI-discovered candidates to exclude them from future discoveries
+- Batch dismiss all unselected candidates at once
+- View dismissed candidates with original AI reasoning and tags
+- Undo dismiss to restore candidates
+- Clear all dismissed candidates for a database
+- Dismissed candidates stored in SQLite with `discovery_dismissed` table
+- API endpoints: POST `/dismiss`, POST `/dismiss-batch`, GET `/dismissed`, DELETE `/dismissed/:personId`, DELETE `/dismissed`
+
+### AI Discovery Debug Endpoints (v0.7.0)
+- New debug endpoints for troubleshooting AI run failures
+- GET `/api/ai-discovery/debug/runs` - list recent AI runs with metadata
+- GET `/api/ai-discovery/debug/runs/:runId` - detailed run info including prompt preview and output
+- Error details capture (last 10 lines of stderr/stdout) stored on failed runs
+
+### Migration Map Visualization (v0.7.x)
+- New 6th tree view mode: "Migration Map" plots ancestors on an interactive world map
+- Leaflet.js map with OpenStreetMap/CartoDB dark tiles
+- Lineage-colored markers (paternal=blue, maternal=red) with generation depth variation
+- Migration polylines connecting parent birth to child birth locations
+- Time range slider for filtering by birth year
+- Paternal/Maternal layer toggle checkboxes
+- Geocoding via Nominatim with permanent SQLite cache (`place_geocode` table)
+- Progressive geocode broadening: strips specific locality parts to resolve historical place names
+- User-initiated batch geocoding with SSE progress streaming (EventSource)
+- Sparse tree map page at `/favorites/sparse-tree/:dbId/map`
+- API: `GET /api/map/:dbId/:personId`, `GET /api/map/:dbId/sparse`, `GET /api/map/geocode/stream`
+
+## Bug Fixes
+
+### CLI Provider Integration (v0.7.0)
+- Fixed large prompt handling via stdin piping for CLI providers (codex, claude-code)
+- Added `-` flag detection in provider args to enable stdin mode
+- Fixed JSON parsing to find LAST valid JSON array in noisy CLI output
+- Handles CLI banners and headers that could interfere with response parsing
+- Improved error capture with `errorDetails` field for debugging failures

PLAN.md

@@ -38,6 +38,7 @@ High-level project roadmap. For detailed phase documentation, see [docs/roadmap.
 | 15.19 | Normalize FamilySearch as downstream provider | ✅ |
 | 15.20 | Relationship linking (parents, spouses, children) | 📋 |
 | 15.22 | Ancestry free hints automation | ✅ |
+| 15.23 | Migration Map visualization | ✅ |
 | 16 | Multi-platform sync architecture | 📋 |
 | 17 | Real-time event system (Socket.IO) | 📋 |
 
@@ -561,6 +562,63 @@ UI Button (ProviderDataTable)
 - `POST /api/ancestry-hints/:dbId/cancel` - Cancel running operation
 - `GET /api/ancestry-hints/status` - Check if running
 
+### Phase 15.23: Migration Map Visualization ✅
+
+A 6th tree view mode plotting ancestors on an interactive Leaflet.js map with lineage-colored markers, parent-child migration lines, time filtering, and layer controls.
+
+**Features:**
+- **Migration Map view mode** - Available in the tree view mode dropdown alongside Fan, Horizontal, Vertical, Columns, Focus
+- **Leaflet.js map** with OpenStreetMap tiles (light) and CartoDB dark tiles (dark theme)
+- **Lineage-colored markers** - Paternal (blue/teal), Maternal (red/coral), Self (purple)
+- **Migration polylines** connecting parent birth locations to child birth locations
+- **Time range slider** filtering ancestors by birth year
+- **Paternal/Maternal layer toggles** for lineage filtering
+- **Auto-fit bounds** on data load
+- **Click popups** with person name, lifespan, places, and link to person detail
+- **Geocoding service** using Nominatim with 1100ms rate limiting and permanent SQLite cache
+- **Geocode progress bar** with SSE streaming for batch geocoding
+- **Sparse tree map page** at `/favorites/sparse-tree/:dbId/map`
+
+**Architecture:**
+```
+Database: place_geocode table (SQLite cache)
+    → Geocode Service (Nominatim + cache)
+    → Map Service (tree data + coordinate joining)
+    → Map Routes (REST + SSE geocode stream)
+    → MigrationMapView (Leaflet + react-leaflet)
+```
+
+**Routes:**
+- `/tree/:dbId/:personId/map` - Ancestry tree migration map
+- `/favorites/sparse-tree/:dbId/map` - Sparse tree (favorites) migration map
+
+**API Endpoints:**
+- `GET /api/map/:dbId/:personId` - Ancestry tree map data
+- `GET /api/map/:dbId/sparse` - Sparse tree map data
+- `GET /api/map/geocode/stream` - Batch geocode via SSE (EventSource)
+- `GET /api/map/geocode/stats` - Geocode cache statistics
+- `POST /api/map/geocode/reset-not-found` - Reset failed entries for retry
+
+**Files Created:**
+- `server/src/db/migrations/006_place_geocode.ts` - Place geocode cache table
+- `server/src/services/geocode.service.ts` - Nominatim geocoding + SQLite cache
+- `server/src/services/map.service.ts` - Map data assembly
+- `server/src/routes/map.routes.ts` - REST + SSE endpoints
+- `client/src/components/ancestry-tree/views/MigrationMapView.tsx` - Leaflet map component
+- `client/src/components/favorites/SparseTreeMapPage.tsx` - Sparse tree map page
+- `client/src/components/map/GeocodeProgressBar.tsx` - Geocode progress UI
+- `client/src/components/map/mapUtils.ts` - Marker/line utilities
+
+**Files Modified:**
+- `server/src/db/migrations/index.ts` - Register migration 006
+- `server/src/db/schema.sql` - Add place_geocode table
+- `server/src/index.ts` - Mount mapRouter
+- `shared/src/index.ts` - MapCoords, MapPerson, MapData, GeocodeProgress types
+- `client/package.json` - leaflet, react-leaflet, @types/leaflet deps
+- `client/src/services/api.ts` - Map API methods
+- `client/src/components/ancestry-tree/AncestryTreeView.tsx` - Add 'map' view mode
+- `client/src/App.tsx` - Sparse tree map route
+
 ### Phase 16: Multi-Platform Sync (Remaining Items)
 
 - ~~Provider cache structure~~ (completed in 15.11)

README.md

@@ -2,6 +2,10 @@
 
 A genealogy toolkit for creating local databases of your family tree, curating favorites, and generating sparse family tree visualizations.
 
+## Origins
+
+This project is a web-based UI enhancement that evolved from the creator's early explorations into learning more about his ancestry using a CLI-based tool: [FamilySearchFinder](https://github.com/atomantic/FamilySearchFinder).
+
 ## What is SparseTree?
 
 SparseTree connects to genealogy providers (FamilySearch, Ancestry, WikiTree, 23andMe) to download and store your ancestry data locally. Once you have your data, you can:

client/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@fsf/client",
-  "version": "0.6.33",
+  "version": "0.7.12",
   "type": "module",
   "scripts": {
     "dev": "vite --port 6373",
@@ -10,17 +10,20 @@
   "dependencies": {
     "@fsf/shared": "*",
     "d3": "^7.9.0",
+    "leaflet": "^1.9.4",
     "lucide-react": "^0.562.0",
     "portos-ai-toolkit": "^0.1.0",
     "react": "^18.3.1",
     "react-dom": "^18.3.1",
     "react-hot-toast": "^2.6.0",
+    "react-leaflet": "^4.2.1",
     "react-router-dom": "^7.1.1",
     "socket.io-client": "^4.8.3",
     "zustand": "^5.0.2"
   },
   "devDependencies": {
     "@types/d3": "^7.4.3",
+    "@types/leaflet": "^1.9.21",
     "@types/react": "^18.3.18",
     "@types/react-dom": "^18.3.5",
     "@vitejs/plugin-react": "^4.3.4",

client/src/App.tsx

@@ -13,6 +13,7 @@ import { BrowserSettingsPage } from './pages/BrowserSettingsPage';
 import { ReportsPage } from './pages/ReportsPage';
 import { FavoritesPage } from './components/favorites/FavoritesPage';
 import { SparseTreePage } from './components/favorites/SparseTreePage';
+import { SparseTreeMapPage } from './components/favorites/SparseTreeMapPage';
 import { DatabaseFavoritesPage } from './components/favorites/DatabaseFavoritesPage';
 import { IntegrityPage } from './components/integrity/IntegrityPage';
 import { AncestryUpdatePage } from './components/ancestry-update';
@@ -41,6 +42,7 @@ function App() {
         <Route path="tools/gedcom" element={<GedcomPage />} />
         <Route path="favorites" element={<FavoritesPage />} />
         <Route path="favorites/sparse-tree/:dbId" element={<SparseTreePage />} />
+        <Route path="favorites/sparse-tree/:dbId/map" element={<SparseTreeMapPage />} />
         <Route path="db/:dbId/favorites" element={<DatabaseFavoritesPage />} />
         <Route path="db/:dbId/integrity" element={<IntegrityPage />} />
         <Route path="db/:dbId/integrity/:tab" element={<IntegrityPage />} />