Docs Refresh & Clean Up

README.md

@@ -1,8 +1,35 @@
 # Forage
 
-Forage is a privacy-first GitHub repository rediscovery tool.
+Forage is a privacy-first GitHub repository rediscovery tool. Users authorize GitHub, import starred repositories into browser-local IndexedDB, run local analysis, review scores/categories, and export results without storing repository data server-side.
 
-The current repo contains planning docs plus a minimal pre-MVP spike app used to validate:
+The current repo contains the real Astro/Svelte web app, Cloudflare Worker auth/import API, shared packages, infrastructure configuration, hosted smoke checks, and the retained pre-MVP spike.
+
+## Documentation Map
+
+Start with [docs/README.md](./docs/README.md) for the full link tree and current hosted environment map.
+
+High-signal docs:
+
+- [Architecture](./docs/02-architecture.md)
+- [Auth And Privacy](./docs/11-auth-and-privacy.md)
+- [Storage Schema](./docs/13-storage-schema.md)
+- [Import Pipeline](./docs/12-import-pipeline.md)
+- [Hosting UI Setup](./docs/21-hosting-ui-setup.md)
+- [Cloudflare Token Permissions](./docs/24-cloudflare-token-permissions.md)
+
+## Current App
+
+The real app is split into:
+
+- `apps/web`: Astro browser app with Svelte islands, IndexedDB storage, local analysis, dashboard, and exports.
+- `apps/worker`: Cloudflare Worker for GitHub auth, session/settings, and GitHub API proxying.
+- `packages/*`: shared contracts, core import state, GitHub normalization, analysis/scoring, and reporting.
+
+Repository data imported through the real app is stored in browser IndexedDB, not in the Worker.
+
+## Pre-MVP Spike
+
+The minimal pre-MVP spike remains as a reference for the first proven vertical slice:
 
 - GitHub App user authorization
 - Authenticated starred repository import
@@ -12,8 +39,6 @@ The current repo contains planning docs plus a minimal pre-MVP spike app used to
 - Browser Web Worker analysis
 - JSON export
 
-## Pre-MVP Spike
-
 Copy `.env.example` to `.env` or export the variables in your shell:
 
 ```sh
@@ -38,11 +63,6 @@ This app is intentionally small and dependency-free. It is not the final Astro/C
 
 ## Real App Local Dev
 
-The real app is split into:
-
-- `apps/web`: Astro browser app
-- `apps/worker`: Cloudflare Worker auth/import API
-
 Copy the Worker env example:
 
 ```sh
@@ -73,8 +93,6 @@ Then open:
 http://127.0.0.1:4321
 ```
 
-Repository data imported through the real app is stored in browser IndexedDB, not in the Worker.
-
 ## Hosted Setup
 
 Hosted testing uses Cloudflare Pages for `apps/web`, a Cloudflare Worker for `apps/worker`, a Durable Object for auth/session coordination, and KV only for small settings records.

docs/02-architecture.md

@@ -44,8 +44,9 @@ Monorepo:
 - pnpm workspaces
 
 apps/
+  pre-mvp/
   web/
-  cli/
+  worker/
 
 packages/
   core/

docs/03-data-model.md

@@ -4,9 +4,9 @@
 Repository metadata should be normalized into a consistent internal model.
 
 Status:
-Discovery Required
+Initial contract implemented
 
-The exact model should be finalized after testing the GitHub API response shape against real accounts. The model below is the starting point, not a locked schema.
+The repository model is implemented in `packages/shared/src/repository.ts` and populated from the GitHub REST starred repository response. The fields below are the current shared contract, with nullable values where GitHub can omit data.
 
 Fields:
 
@@ -61,8 +61,8 @@ Category Metadata:
 - repo name
 
 Required follow-up:
-- Confirm available fields from GitHub's starred repository endpoint.
-- Decide whether per-repository detail calls are needed beyond the starred list response.
-- Define nullable fields.
-- Define IndexedDB schema and migration behavior.
-- Define analysis result and export models separately from repository metadata.
+- Continue validating nullable fields against larger real account imports.
+- Avoid per-repository detail calls unless a specific MVP feature requires fields missing from the starred list response.
+- Keep IndexedDB schema and migration behavior aligned with [Storage Schema](./13-storage-schema.md).
+- Keep analysis result and export models separate from repository metadata.
+- Update this document and `packages/shared` together when repository fields change.

docs/04-categories.md

@@ -4,7 +4,7 @@
 Categories are fully configuration-driven.
 
 Status:
-First v0.1 weighted-rule model drafted
+First v0.1 weighted-rule model implemented
 
 MVP category strategy:
 Start broad and general. The first category set should be wide enough to classify most repositories without getting stuck on highly specific niches.
@@ -59,10 +59,10 @@ languages:
 
 Open design items:
 - Tune category coverage against real imports
-- Decide when to persist analysis results versus calculating on read
-- Multi-category behavior
+- Continue validating persisted analysis results versus calculating on read for future schema changes
+- Tune multi-category behavior and display priority
 - Deprecated category handling
-- Versioning and migration strategy for category changes
+- Versioning and migration strategy for future category changes
 - Criteria for promoting narrower categories after real data review
 
 Current v0.1 behavior:

docs/05-scoring-and-insights.md

@@ -2,7 +2,7 @@
 # Scoring & Insights
 
 Status:
-First v0.1 foundational scoring model drafted
+First v0.1 foundational scoring model implemented
 
 Purpose:
 Fun, explainable ranking system.
@@ -55,8 +55,8 @@ Formula requirements:
 - Tune weights for each signal.
 - Tune caps and normalization rules so very popular repositories do not dominate every ranking.
 - Tune how archived, disabled, stale, and recently revived repositories affect scores.
-- Define how score version changes affect existing local analysis.
-- Define how user-agnostic scores feed, but do not mutate, future personalized match scoring.
+- Keep score version changes explicit in local analysis results and exports.
+- Keep user-agnostic scores separate from any future personalized match scoring.
 
 Current v0.1 scoring behavior:
 - Score version: `foundational-v0.1.0`.

docs/06-pwa-and-storage.md

@@ -1,12 +1,21 @@
 
 # PWA & Storage Strategy
 
+Status:
+Initial local-first storage implemented
+
 Primary storage location:
 
 Browser
 
 Technology:
 - IndexedDB
+- Dexie in `apps/web/src/lib/db-schema.ts`
+
+Current database:
+- Name: `forage`
+- Version: `3`
+- Stores: `repositories`, `importEvents`, `metadata`, `analysisResults`
 
 Stored locally:
 
@@ -42,14 +51,18 @@ Import State
 → Restore application state
 
 Open storage decisions:
-- IndexedDB store names and schema versions
-- Migration strategy
 - Local data reset behavior
 - Import/export merge behavior
 - Browser quota handling
 - Multi-tab safety
 - Offline state and stale data indicators
 - Whether category config ships with the app, syncs from the server, or both
 
+Implemented storage decisions:
+- Repository records, import events, analysis results, local library profile, and local operation lock metadata live in browser IndexedDB.
+- Server settings are limited to small preferences and analytics opt-in state.
+- The current test suite includes a fixture-backed migration check from the previous v2 local schema into v3.
+- Local reset clears browser-local repository and analysis data without requiring server-side repository deletion because repository data is never uploaded.
+
 Migration posture:
 Use IndexedDB schema migrations when practical. If a future local schema migration would be disproportionately complex, the app may require a re-import from GitHub, but that should be treated as a fallback rather than the default strategy.

docs/07-workers-and-processing.md

@@ -4,7 +4,7 @@
 Import Web Worker
 
 Status:
-Planned
+Planned for pagination/orchestration hardening. Current imports are initiated from the web app and use shared import state contracts.
 
 Responsibilities:
 - GitHub pagination
@@ -40,7 +40,7 @@ Cloudflare Workers are separate from these browser workers and should only handl
 
 Required follow-up:
 - Move import pagination/orchestration into a browser worker.
-- Decide REST vs GraphQL for import.
-- Confirm pagination and rate-limit behavior.
+- Keep REST as the MVP import path; revisit GraphQL only if future richer metadata needs justify it.
+- Continue measuring pagination and rate-limit behavior on hosted imports.
 - Decide incremental refresh behavior.
 - Decide how partial imports are represented.

docs/08-dashboard-and-ux.md

@@ -32,22 +32,27 @@ Insight Panels
 Exports
 - JSON
 - CSV
-- Markdown
-- HTML
+
+Current implemented states:
+- Authenticated and disconnected sessions
+- Local library owner messaging
+- Initial loading/hydration
+- Import progress
+- Import cancellation
+- Rate-limit terminal state
+- No matching results
+- Local data reset confirmation
+- JSON and CSV export actions
 
 Future:
-More guided exploration experiences.
+More guided exploration experiences, Markdown export, and HTML report export.
 
 Deferred but required UX states:
 - Empty state
 - GitHub auth failure
-- Import progress
-- Import cancellation
 - Rate limit warning
 - Partial import recovery
-- No matching results
 - Export success and failure
 - Offline or stale data state
-- Local data reset confirmation
 
 MVP UX can stay utilitarian at first, but these states should exist before a public release.

docs/09-roadmap.md

@@ -1,58 +1,39 @@
 
 # Roadmap
 
-Phase 0
-- Planning
-- Architecture
-- Repository Setup
-- GitHub App auth/privacy design
-- GitHub API spike
-- Import/storage spike
-
-Phase 1
-- Core Engine
-- Models
-- Categorization
-- Scoring
-- Reporting
-
-Phase 2
-- CLI Harness
-- Testing
-- Local Analysis
-
-Phase 3
-- Run Against Real GitHub Accounts
-- Refine Categories
-- Refine Insights
-
-Phase 4
-- Astro Application
-- GitHub authorization
-- Cloudflare Infrastructure
-
-Phase 5
-- Dashboard Experience
-- Search
-- Filters
-- Exports
-
-Phase 6
-- Analytics
-- Preferences
-- Opt-Out Support
+Status:
+Current MVP hardening roadmap
+
+Completed:
+- Planning, architecture, and repository setup.
+- GitHub App auth/privacy design.
+- Pre-MVP GitHub API import and IndexedDB storage spike.
+- Real Astro/Svelte web app scaffold.
+- Cloudflare Worker auth/session/settings/API boundary.
+- Shared contracts and package structure.
+- REST starred repository import, normalization, local storage, and import events.
+- Category matching, user-agnostic scoring, labels, and analysis details.
+- Dashboard summary, search, filters, sorting, details, JSON export, and CSV export.
+- CI, coverage, formatting, type checks, builds, OpenTofu format checks, and hosted smoke script.
+- Cloudflare hosting/security docs, OpenTofu configuration, staging Access posture, and deploy workflow.
+
+Current:
+- Import resilience and recovery UX.
+- First-run, reconnect, and local-library ownership UX.
+- Hosted staging verification and production deploy rehearsal.
+- Real account analysis review for category and score tuning.
+- Security/privacy regression checks before broader testing.
+
+Next:
+- Production GitHub App and Cloudflare secret/binding verification.
+- Browser smoke tests for auth/import/dashboard/export flows.
+- Settings/privacy screen refinement.
+- Export/restore compatibility decisions.
+- Browser quota and multi-tab edge case handling.
+- Broader beta acceptance pass against [MVP Acceptance Criteria](./14-mvp-acceptance-criteria.md).
 
 Future
 - Saved Views
 - Category Requests
 - GitHub Action
 - AI-assisted features (optional)
-
-Recommended first technical spike:
-- Create GitHub App authorization proof of concept.
-- Fetch starred repositories for the authenticated user.
-- Preserve starred_at when available.
-- Store normalized records in IndexedDB.
-- Record import event metadata.
-- Run a minimal browser analysis worker.
-- Export JSON.

docs/10-open-questions.md

@@ -3,16 +3,15 @@
 
 Items intentionally deferred.
 
-- Exact scoring formulas
-- Category weighting strategy
-- Insight calculation algorithms
+- Scoring and category tuning after reviewing more real imports
+- Insight calculation algorithms beyond the current analysis detail view
 - Future personalized Match Score design
-- Export format enhancements
+- Export format enhancements beyond JSON and CSV
 - Category submission workflow
-- Exact GitHub fields used in the canonical model
-- IndexedDB schema and migration strategy
-- UI details beyond the first functional dashboard
-- Exact production hostnames
+- Restore/import from a previous Forage export
+- Browser quota warnings and offline/stale data behavior
+- Multi-tab import/reset conflict handling beyond the current local operation lock
+- Pages preview access policy for non-canonical preview URLs
 
 Current philosophy:
 
@@ -31,5 +30,7 @@ Repository scores are user-agnostic. Any future preference-aware ranking should
 Hosting/security direction:
 Use Cloudflare Pages for the static web app and a separate Cloudflare Worker for auth/session/settings/API proxying. See [Hosting And Security Plan](./20-hosting-and-security.md).
 
+Current hosted domains are tracked in [Documentation](./README.md) and [Hosting UI Setup](./21-hosting-ui-setup.md).
+
 MVP token/session direction:
 Use expiring GitHub App user access tokens and require reconnect after session expiry. Do not persist refresh tokens for MVP. Minimal server-side account state can be deleted through the Worker account deletion endpoint.

docs/12-import-pipeline.md

@@ -12,13 +12,13 @@ Each import is recorded as an import event. Repository data from that event is s
 Initial source:
 GitHub REST endpoint for repositories starred by the authenticated user.
 
-Important API behavior to verify:
-- Pagination with up to 100 records per page
-- Rate limits and secondary rate limits
-- Starred timestamp support
-- Whether all required repository fields are present in the starred list response
-- Whether additional repository detail calls are needed
-- Whether private starred repositories appear with the selected authorization model
+Known API behavior:
+- The current MVP import uses the GitHub REST starred repositories endpoint through the Worker GitHub API proxy.
+- Pagination uses up to 100 records per page.
+- Starred timestamp support is available and preserved as `starred_at`.
+- The starred list response has been sufficient for the current repository model, category matching, scoring, dashboard, and export paths.
+- Per-repository detail calls are avoided for MVP to reduce latency and rate-limit pressure.
+- Rate limits and secondary rate limits still need real hosted testing across more accounts.
 
 Import stages:
 - Start authenticated import
@@ -56,8 +56,8 @@ Rate-limit posture:
 - Keep refresh user-initiated at first, then consider scheduled/background refresh only after measuring API behavior.
 
 Open implementation decisions:
-- REST vs GraphQL after the initial spike
 - Full reimport is the MVP refresh model; completed imports reconcile local data to remove repositories that are no longer starred.
 - Whether to keep previous analysis when metadata changes
 - Partial import recovery format
 - Import metadata stored per run
+- Whether GraphQL is worth revisiting for future richer metadata after MVP