v0.7.x: Migration Map, AI Discovery Dismiss, Debug Endpoints

[atomantic]

Summary

• Migration Map Visualization — New 6th tree view mode plotting ancestors on an interactive Leaflet.js world map with lineage-colored markers, migration polylines, time range filtering, and paternal/maternal layer toggles
• AI Discovery Dismiss — Dismiss AI-discovered candidates individually or in batch, view/restore dismissed candidates, clear all dismissed for a database
• AI Discovery Debug Endpoints — Inspect recent AI runs, view prompts, outputs, and error details for failed runs
• Geocoding Service — Nominatim integration with permanent SQLite cache, progressive broadening for historical place names, SSE-streamed batch geocoding via EventSource

New Features

Migration Map (Phase 15.23)

• Leaflet.js map with OpenStreetMap (light) and CartoDB (dark) tile layers
• Lineage-colored markers: paternal (blue/teal), maternal (red/coral), self (purple)
• Migration polylines connecting parent birth → child birth locations
• Time range slider filtering ancestors by birth year
• Paternal/Maternal layer toggle checkboxes
• Auto-fit bounds on data load with click popups linking to person detail
• Sparse tree map page at /favorites/sparse-tree/:dbId/map
• Progressive geocode broadening: strips specific locality parts to resolve historical place names (e.g. "Cornouaile, Visseiche, Ille-et-Vilaine, Brittany, France" → "Visseiche, Ille-et-Vilaine, Brittany, France")

AI Discovery Improvements

• Dismiss/batch-dismiss/undo-dismiss/clear-dismissed functionality
• discovery_dismissed SQLite table
• Debug endpoints: GET /api/ai-discovery/debug/runs, GET /api/ai-discovery/debug/runs/:runId
• Error details capture (last 10 lines stderr/stdout) on failed runs
• Fixed large prompt handling via stdin piping for CLI providers

API Endpoints

Method	Path	Description
GET	/api/map/:dbId/:personId	Ancestry tree map data
GET	/api/map/:dbId/sparse	Sparse tree map data
GET	/api/map/geocode/stream?dbId=	SSE batch geocode stream
GET	/api/map/geocode/stats	Geocode cache statistics
POST	/api/map/geocode/reset-not-found	Reset failed entries for retry
POST	/api/ai-discovery/:dbId/dismiss	Dismiss a candidate
POST	/api/ai-discovery/:dbId/dismiss-batch	Batch dismiss candidates
GET	/api/ai-discovery/:dbId/dismissed	List dismissed candidates
DELETE	/api/ai-discovery/:dbId/dismissed/:personId	Undo dismiss
DELETE	/api/ai-discovery/:dbId/dismissed	Clear all dismissed
GET	/api/ai-discovery/debug/runs	List recent AI runs
GET	/api/ai-discovery/debug/runs/:runId	Run detail with prompt/output

Files Changed

New files (10): migrations (005, 006), geocode service, map service, map routes, MigrationMapView, SparseTreeMapPage, GeocodeProgressBar, mapUtils

Modified files (19): schema.sql, migrations index, server index, shared types, client api, AncestryTreeView, SparseTreePage, App.tsx, ai-discovery routes/service/modal, package files, changelog, PLAN.md, README

Test Plan

• Navigate to any person's tree view → select "Migration Map" from view mode dropdown
• Verify map renders with tiles and auto-fits bounds
• Click "Geocode Places" → verify SSE progress streams all places
• Verify markers appear with correct lineage colors
• Verify migration polylines connect parent → child birth locations
• Test time range slider and paternal/maternal toggles
• Test dark/light theme toggle
• Navigate to Favorites → Sparse Tree → Map link
• Verify AI Discovery dismiss, batch dismiss, undo, clear all
• Verify AI Discovery debug endpoints return run data

[Copilot]

Pull request overview

This PR introduces a comprehensive migration map visualization feature alongside AI discovery improvements and debugging capabilities. The migration map is a new tree view mode that plots ancestors geographically on an interactive Leaflet.js map with lineage-colored markers, migration polylines, time-based filtering, and geocoding via Nominatim. Additionally, the PR adds functionality to dismiss AI discovery candidates, debug endpoints for troubleshooting AI runs, and integrates the AI toolkit service for better provider management.

Changes:

• Migration Map visualization with Leaflet.js, including geocoding service with Nominatim integration and permanent SQLite cache
• AI Discovery dismiss/restore functionality with new discovery_dismissed table
• AI Discovery debug endpoints exposing run metadata, prompts, and outputs
• Progressive geocoding with broadening strategy for historical place names
• SSE-based batch geocoding with EventSource for real-time progress

Reviewed changes

Copilot reviewed 28 out of 29 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
server/src/db/migrations/005_discovery_dismissed.ts	New table for tracking dismissed AI candidates
server/src/db/migrations/006_place_geocode.ts	New table for geocoding cache
server/src/services/geocode.service.ts	Nominatim integration with rate limiting and caching
server/src/services/map.service.ts	Map data assembly joining tree data with geocoded coordinates
server/src/routes/map.routes.ts	REST and SSE endpoints for map data and geocoding
server/src/services/ai-discovery.service.ts	AI toolkit integration and dismiss functionality
server/src/routes/ai-discovery.routes.ts	Debug endpoints and dismiss operations
client/src/components/ancestry-tree/views/MigrationMapView.tsx	Leaflet map component with filtering and controls
client/src/components/map/GeocodeProgressBar.tsx	SSE progress UI for batch geocoding
client/src/components/map/mapUtils.ts	Marker and popup HTML generation utilities
client/src/components/favorites/SparseTreeMapPage.tsx	Standalone map page for favorites
client/src/components/ai/AiDiscoveryModal.tsx	Dismiss UI and generation filtering
shared/src/index.ts	Type definitions for map data structures
client/src/services/api.ts	API methods for map and dismiss operations
package.json files	Version bumps and new dependencies (leaflet, react-leaflet)

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 28 out of 29 changed files in this pull request and generated 13 comments.

Comments suppressed due to low confidence (1)

server/src/services/ai-discovery.service.ts:460

• maxGenerations is applied only in the SQLite query branch. In the fallback branch (!databaseService.isSqliteEnabled()), the filter is ignored, so requests with maxGenerations won’t behave consistently depending on backend mode. Consider applying a generation filter in the fallback path too (if generation info is available), or rejecting/ignoring the option consistently when SQLite is disabled.

    } else {
      // Fallback to loading all
      const db = await databaseService.getDatabase(dbId);
      personsToAnalyze = Object.entries(db)
        .filter(([id, person]) => {
          if (excludeIds.has(id)) return false;
          if (minBirthYear !== undefined) {
            const birthYear = getBirthYear(person);
            if (birthYear !== null && birthYear < minBirthYear) return false;
          }
          return true;
        })
        .slice(0, sampleSize)
        .map(([id, person]) => ({ id, person }));
    }

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 28 out of 29 changed files in this pull request and generated 7 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 28 out of 29 changed files in this pull request and generated 5 comments.

Comments suppressed due to low confidence (2)

server/src/routes/ai-discovery.routes.ts:35

• The sampleSize parameter in the quick discovery endpoint is not validated on the server side. A malicious client could send an extremely large value (e.g., millions) which could cause performance issues or resource exhaustion when querying the database and sending large prompts to the AI provider. Consider adding validation to ensure sampleSize is within reasonable bounds (e.g., max 10000) and return a 400 error if exceeded.

router.post('/:dbId/quick', async (req: Request, res: Response) => {
  const { dbId } = req.params;
  const { sampleSize, model, excludeBiblical, minBirthYear, maxGenerations, customPrompt } = req.body;

  logger.start('ai-discovery', `Quick discovery request dbId=${dbId} sample=${sampleSize || 100} model=${model || 'default'} excludeBiblical=${excludeBiblical || false} maxGenerations=${maxGenerations || 'all'} prompt=${customPrompt ? `"${customPrompt.slice(0, 50)}..."` : 'none'}`);

  const result = await aiDiscoveryService.quickDiscovery(dbId, {
    sampleSize: sampleSize || 100,
    model,
    excludeBiblical: excludeBiblical || false,
    minBirthYear,
    maxGenerations,
    customPrompt,
  }).catch(err => {
    logger.error('ai-discovery', `Quick discovery failed: ${err.message}`);
    res.status(500).json({ success: false, error: err.message });
    return null;
  });

  if (result !== null) {
    logger.done('ai-discovery', `Quick discovery complete: analyzed=${result.totalAnalyzed} candidates=${result.candidates.length}`);
    res.json({ success: true, data: result });
  }
});

server/src/routes/ai-discovery.routes.ts:35

• The maxGenerations parameter is not validated on the server side. While the client limits it to 100, a malicious client could send an arbitrarily large value or negative number. Consider adding server-side validation to ensure maxGenerations is within reasonable bounds (e.g., 1-100) if provided.

router.post('/:dbId/quick', async (req: Request, res: Response) => {
  const { dbId } = req.params;
  const { sampleSize, model, excludeBiblical, minBirthYear, maxGenerations, customPrompt } = req.body;

  logger.start('ai-discovery', `Quick discovery request dbId=${dbId} sample=${sampleSize || 100} model=${model || 'default'} excludeBiblical=${excludeBiblical || false} maxGenerations=${maxGenerations || 'all'} prompt=${customPrompt ? `"${customPrompt.slice(0, 50)}..."` : 'none'}`);

  const result = await aiDiscoveryService.quickDiscovery(dbId, {
    sampleSize: sampleSize || 100,
    model,
    excludeBiblical: excludeBiblical || false,
    minBirthYear,
    maxGenerations,
    customPrompt,
  }).catch(err => {
    logger.error('ai-discovery', `Quick discovery failed: ${err.message}`);
    res.status(500).json({ success: false, error: err.message });
    return null;
  });

  if (result !== null) {
    logger.done('ai-discovery', `Quick discovery complete: analyzed=${result.totalAnalyzed} candidates=${result.candidates.length}`);
    res.json({ success: true, data: result });
  }
});

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 28 out of 29 changed files in this pull request and generated 6 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 28 out of 29 changed files in this pull request and generated 6 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 28 out of 29 changed files in this pull request and generated 7 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Pull request overview

Copilot reviewed 28 out of 29 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 28 out of 29 changed files in this pull request and generated 4 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 28 out of 29 changed files in this pull request and generated 4 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.