From 67059a68d1df2be397998431b62b2f3b339aae1d Mon Sep 17 00:00:00 2001 From: Dylan Steele Date: Wed, 10 Jun 2026 21:09:04 -0400 Subject: [PATCH] Refresh docs current state and navigation --- README.md | 40 +++++++++++---- docs/02-architecture.md | 3 +- docs/03-data-model.md | 14 ++--- docs/04-categories.md | 8 +-- docs/05-scoring-and-insights.md | 6 +-- docs/06-pwa-and-storage.md | 17 ++++++- docs/07-workers-and-processing.md | 6 +-- docs/08-dashboard-and-ux.md | 19 ++++--- docs/09-roadmap.md | 79 +++++++++++------------------ docs/10-open-questions.md | 17 ++++--- docs/12-import-pipeline.md | 16 +++--- docs/16-implementation-readiness.md | 48 +++++++++++------- docs/17-ci-and-quality-gates.md | 2 +- docs/20-hosting-and-security.md | 30 ++++++----- docs/21-hosting-ui-setup.md | 53 ++++++++++--------- docs/23-deployment-automation.md | 8 +-- docs/README.md | 28 ++++++++-- 17 files changed, 228 insertions(+), 166 deletions(-) diff --git a/README.md b/README.md index 9b22713..8865b96 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,35 @@ # Forage -Forage is a privacy-first GitHub repository rediscovery tool. +Forage is a privacy-first GitHub repository rediscovery tool. Users authorize GitHub, import starred repositories into browser-local IndexedDB, run local analysis, review scores/categories, and export results without storing repository data server-side. -The current repo contains planning docs plus a minimal pre-MVP spike app used to validate: +The current repo contains the real Astro/Svelte web app, Cloudflare Worker auth/import API, shared packages, infrastructure configuration, hosted smoke checks, and the retained pre-MVP spike. + +## Documentation Map + +Start with [docs/README.md](./docs/README.md) for the full link tree and current hosted environment map. + +High-signal docs: + +- [Architecture](./docs/02-architecture.md) +- [Auth And Privacy](./docs/11-auth-and-privacy.md) +- [Storage Schema](./docs/13-storage-schema.md) +- [Import Pipeline](./docs/12-import-pipeline.md) +- [Hosting UI Setup](./docs/21-hosting-ui-setup.md) +- [Cloudflare Token Permissions](./docs/24-cloudflare-token-permissions.md) + +## Current App + +The real app is split into: + +- `apps/web`: Astro browser app with Svelte islands, IndexedDB storage, local analysis, dashboard, and exports. +- `apps/worker`: Cloudflare Worker for GitHub auth, session/settings, and GitHub API proxying. +- `packages/*`: shared contracts, core import state, GitHub normalization, analysis/scoring, and reporting. + +Repository data imported through the real app is stored in browser IndexedDB, not in the Worker. + +## Pre-MVP Spike + +The minimal pre-MVP spike remains as a reference for the first proven vertical slice: - GitHub App user authorization - Authenticated starred repository import @@ -12,8 +39,6 @@ The current repo contains planning docs plus a minimal pre-MVP spike app used to - Browser Web Worker analysis - JSON export -## Pre-MVP Spike - Copy `.env.example` to `.env` or export the variables in your shell: ```sh @@ -38,11 +63,6 @@ This app is intentionally small and dependency-free. It is not the final Astro/C ## Real App Local Dev -The real app is split into: - -- `apps/web`: Astro browser app -- `apps/worker`: Cloudflare Worker auth/import API - Copy the Worker env example: ```sh @@ -73,8 +93,6 @@ Then open: http://127.0.0.1:4321 ``` -Repository data imported through the real app is stored in browser IndexedDB, not in the Worker. - ## Hosted Setup Hosted testing uses Cloudflare Pages for `apps/web`, a Cloudflare Worker for `apps/worker`, a Durable Object for auth/session coordination, and KV only for small settings records. diff --git a/docs/02-architecture.md b/docs/02-architecture.md index 5782a0a..90f68b2 100644 --- a/docs/02-architecture.md +++ b/docs/02-architecture.md @@ -44,8 +44,9 @@ Monorepo: - pnpm workspaces apps/ + pre-mvp/ web/ - cli/ + worker/ packages/ core/ diff --git a/docs/03-data-model.md b/docs/03-data-model.md index 11f15c9..9e1ff2a 100644 --- a/docs/03-data-model.md +++ b/docs/03-data-model.md @@ -4,9 +4,9 @@ Repository metadata should be normalized into a consistent internal model. Status: -Discovery Required +Initial contract implemented -The exact model should be finalized after testing the GitHub API response shape against real accounts. The model below is the starting point, not a locked schema. +The repository model is implemented in `packages/shared/src/repository.ts` and populated from the GitHub REST starred repository response. The fields below are the current shared contract, with nullable values where GitHub can omit data. Fields: @@ -61,8 +61,8 @@ Category Metadata: - repo name Required follow-up: -- Confirm available fields from GitHub's starred repository endpoint. -- Decide whether per-repository detail calls are needed beyond the starred list response. -- Define nullable fields. -- Define IndexedDB schema and migration behavior. -- Define analysis result and export models separately from repository metadata. +- Continue validating nullable fields against larger real account imports. +- Avoid per-repository detail calls unless a specific MVP feature requires fields missing from the starred list response. +- Keep IndexedDB schema and migration behavior aligned with [Storage Schema](./13-storage-schema.md). +- Keep analysis result and export models separate from repository metadata. +- Update this document and `packages/shared` together when repository fields change. diff --git a/docs/04-categories.md b/docs/04-categories.md index 6d4fa22..e060400 100644 --- a/docs/04-categories.md +++ b/docs/04-categories.md @@ -4,7 +4,7 @@ Categories are fully configuration-driven. Status: -First v0.1 weighted-rule model drafted +First v0.1 weighted-rule model implemented MVP category strategy: Start broad and general. The first category set should be wide enough to classify most repositories without getting stuck on highly specific niches. @@ -59,10 +59,10 @@ languages: Open design items: - Tune category coverage against real imports -- Decide when to persist analysis results versus calculating on read -- Multi-category behavior +- Continue validating persisted analysis results versus calculating on read for future schema changes +- Tune multi-category behavior and display priority - Deprecated category handling -- Versioning and migration strategy for category changes +- Versioning and migration strategy for future category changes - Criteria for promoting narrower categories after real data review Current v0.1 behavior: diff --git a/docs/05-scoring-and-insights.md b/docs/05-scoring-and-insights.md index 6c0c161..d343fd0 100644 --- a/docs/05-scoring-and-insights.md +++ b/docs/05-scoring-and-insights.md @@ -2,7 +2,7 @@ # Scoring & Insights Status: -First v0.1 foundational scoring model drafted +First v0.1 foundational scoring model implemented Purpose: Fun, explainable ranking system. @@ -55,8 +55,8 @@ Formula requirements: - Tune weights for each signal. - Tune caps and normalization rules so very popular repositories do not dominate every ranking. - Tune how archived, disabled, stale, and recently revived repositories affect scores. -- Define how score version changes affect existing local analysis. -- Define how user-agnostic scores feed, but do not mutate, future personalized match scoring. +- Keep score version changes explicit in local analysis results and exports. +- Keep user-agnostic scores separate from any future personalized match scoring. Current v0.1 scoring behavior: - Score version: `foundational-v0.1.0`. diff --git a/docs/06-pwa-and-storage.md b/docs/06-pwa-and-storage.md index b96fedf..c5baeeb 100644 --- a/docs/06-pwa-and-storage.md +++ b/docs/06-pwa-and-storage.md @@ -1,12 +1,21 @@ # PWA & Storage Strategy +Status: +Initial local-first storage implemented + Primary storage location: Browser Technology: - IndexedDB +- Dexie in `apps/web/src/lib/db-schema.ts` + +Current database: +- Name: `forage` +- Version: `3` +- Stores: `repositories`, `importEvents`, `metadata`, `analysisResults` Stored locally: @@ -42,8 +51,6 @@ Import State → Restore application state Open storage decisions: -- IndexedDB store names and schema versions -- Migration strategy - Local data reset behavior - Import/export merge behavior - Browser quota handling @@ -51,5 +58,11 @@ Open storage decisions: - Offline state and stale data indicators - Whether category config ships with the app, syncs from the server, or both +Implemented storage decisions: +- Repository records, import events, analysis results, local library profile, and local operation lock metadata live in browser IndexedDB. +- Server settings are limited to small preferences and analytics opt-in state. +- The current test suite includes a fixture-backed migration check from the previous v2 local schema into v3. +- Local reset clears browser-local repository and analysis data without requiring server-side repository deletion because repository data is never uploaded. + Migration posture: Use IndexedDB schema migrations when practical. If a future local schema migration would be disproportionately complex, the app may require a re-import from GitHub, but that should be treated as a fallback rather than the default strategy. diff --git a/docs/07-workers-and-processing.md b/docs/07-workers-and-processing.md index 8fea585..2071cf4 100644 --- a/docs/07-workers-and-processing.md +++ b/docs/07-workers-and-processing.md @@ -4,7 +4,7 @@ Import Web Worker Status: -Planned +Planned for pagination/orchestration hardening. Current imports are initiated from the web app and use shared import state contracts. Responsibilities: - GitHub pagination @@ -40,7 +40,7 @@ Cloudflare Workers are separate from these browser workers and should only handl Required follow-up: - Move import pagination/orchestration into a browser worker. -- Decide REST vs GraphQL for import. -- Confirm pagination and rate-limit behavior. +- Keep REST as the MVP import path; revisit GraphQL only if future richer metadata needs justify it. +- Continue measuring pagination and rate-limit behavior on hosted imports. - Decide incremental refresh behavior. - Decide how partial imports are represented. diff --git a/docs/08-dashboard-and-ux.md b/docs/08-dashboard-and-ux.md index ccf5c56..27722b3 100644 --- a/docs/08-dashboard-and-ux.md +++ b/docs/08-dashboard-and-ux.md @@ -32,22 +32,27 @@ Insight Panels Exports - JSON - CSV -- Markdown -- HTML + +Current implemented states: +- Authenticated and disconnected sessions +- Local library owner messaging +- Initial loading/hydration +- Import progress +- Import cancellation +- Rate-limit terminal state +- No matching results +- Local data reset confirmation +- JSON and CSV export actions Future: -More guided exploration experiences. +More guided exploration experiences, Markdown export, and HTML report export. Deferred but required UX states: - Empty state - GitHub auth failure -- Import progress -- Import cancellation - Rate limit warning - Partial import recovery -- No matching results - Export success and failure - Offline or stale data state -- Local data reset confirmation MVP UX can stay utilitarian at first, but these states should exist before a public release. diff --git a/docs/09-roadmap.md b/docs/09-roadmap.md index 8ed6a1c..bf78cc5 100644 --- a/docs/09-roadmap.md +++ b/docs/09-roadmap.md @@ -1,58 +1,39 @@ # Roadmap -Phase 0 -- Planning -- Architecture -- Repository Setup -- GitHub App auth/privacy design -- GitHub API spike -- Import/storage spike - -Phase 1 -- Core Engine -- Models -- Categorization -- Scoring -- Reporting - -Phase 2 -- CLI Harness -- Testing -- Local Analysis - -Phase 3 -- Run Against Real GitHub Accounts -- Refine Categories -- Refine Insights - -Phase 4 -- Astro Application -- GitHub authorization -- Cloudflare Infrastructure - -Phase 5 -- Dashboard Experience -- Search -- Filters -- Exports - -Phase 6 -- Analytics -- Preferences -- Opt-Out Support +Status: +Current MVP hardening roadmap + +Completed: +- Planning, architecture, and repository setup. +- GitHub App auth/privacy design. +- Pre-MVP GitHub API import and IndexedDB storage spike. +- Real Astro/Svelte web app scaffold. +- Cloudflare Worker auth/session/settings/API boundary. +- Shared contracts and package structure. +- REST starred repository import, normalization, local storage, and import events. +- Category matching, user-agnostic scoring, labels, and analysis details. +- Dashboard summary, search, filters, sorting, details, JSON export, and CSV export. +- CI, coverage, formatting, type checks, builds, OpenTofu format checks, and hosted smoke script. +- Cloudflare hosting/security docs, OpenTofu configuration, staging Access posture, and deploy workflow. + +Current: +- Import resilience and recovery UX. +- First-run, reconnect, and local-library ownership UX. +- Hosted staging verification and production deploy rehearsal. +- Real account analysis review for category and score tuning. +- Security/privacy regression checks before broader testing. + +Next: +- Production GitHub App and Cloudflare secret/binding verification. +- Browser smoke tests for auth/import/dashboard/export flows. +- Settings/privacy screen refinement. +- Export/restore compatibility decisions. +- Browser quota and multi-tab edge case handling. +- Broader beta acceptance pass against [MVP Acceptance Criteria](./14-mvp-acceptance-criteria.md). Future - Saved Views - Category Requests - GitHub Action - AI-assisted features (optional) - -Recommended first technical spike: -- Create GitHub App authorization proof of concept. -- Fetch starred repositories for the authenticated user. -- Preserve starred_at when available. -- Store normalized records in IndexedDB. -- Record import event metadata. -- Run a minimal browser analysis worker. -- Export JSON. diff --git a/docs/10-open-questions.md b/docs/10-open-questions.md index a96e5ed..04a6814 100644 --- a/docs/10-open-questions.md +++ b/docs/10-open-questions.md @@ -3,16 +3,15 @@ Items intentionally deferred. -- Exact scoring formulas -- Category weighting strategy -- Insight calculation algorithms +- Scoring and category tuning after reviewing more real imports +- Insight calculation algorithms beyond the current analysis detail view - Future personalized Match Score design -- Export format enhancements +- Export format enhancements beyond JSON and CSV - Category submission workflow -- Exact GitHub fields used in the canonical model -- IndexedDB schema and migration strategy -- UI details beyond the first functional dashboard -- Exact production hostnames +- Restore/import from a previous Forage export +- Browser quota warnings and offline/stale data behavior +- Multi-tab import/reset conflict handling beyond the current local operation lock +- Pages preview access policy for non-canonical preview URLs Current philosophy: @@ -31,5 +30,7 @@ Repository scores are user-agnostic. Any future preference-aware ranking should Hosting/security direction: Use Cloudflare Pages for the static web app and a separate Cloudflare Worker for auth/session/settings/API proxying. See [Hosting And Security Plan](./20-hosting-and-security.md). +Current hosted domains are tracked in [Documentation](./README.md) and [Hosting UI Setup](./21-hosting-ui-setup.md). + MVP token/session direction: Use expiring GitHub App user access tokens and require reconnect after session expiry. Do not persist refresh tokens for MVP. Minimal server-side account state can be deleted through the Worker account deletion endpoint. diff --git a/docs/12-import-pipeline.md b/docs/12-import-pipeline.md index f204328..2b7195b 100644 --- a/docs/12-import-pipeline.md +++ b/docs/12-import-pipeline.md @@ -12,13 +12,13 @@ Each import is recorded as an import event. Repository data from that event is s Initial source: GitHub REST endpoint for repositories starred by the authenticated user. -Important API behavior to verify: -- Pagination with up to 100 records per page -- Rate limits and secondary rate limits -- Starred timestamp support -- Whether all required repository fields are present in the starred list response -- Whether additional repository detail calls are needed -- Whether private starred repositories appear with the selected authorization model +Known API behavior: +- The current MVP import uses the GitHub REST starred repositories endpoint through the Worker GitHub API proxy. +- Pagination uses up to 100 records per page. +- Starred timestamp support is available and preserved as `starred_at`. +- The starred list response has been sufficient for the current repository model, category matching, scoring, dashboard, and export paths. +- Per-repository detail calls are avoided for MVP to reduce latency and rate-limit pressure. +- Rate limits and secondary rate limits still need real hosted testing across more accounts. Import stages: - Start authenticated import @@ -56,8 +56,8 @@ Rate-limit posture: - Keep refresh user-initiated at first, then consider scheduled/background refresh only after measuring API behavior. Open implementation decisions: -- REST vs GraphQL after the initial spike - Full reimport is the MVP refresh model; completed imports reconcile local data to remove repositories that are no longer starred. - Whether to keep previous analysis when metadata changes - Partial import recovery format - Import metadata stored per run +- Whether GraphQL is worth revisiting for future richer metadata after MVP diff --git a/docs/16-implementation-readiness.md b/docs/16-implementation-readiness.md index 9b412b9..eccd01c 100644 --- a/docs/16-implementation-readiness.md +++ b/docs/16-implementation-readiness.md @@ -1,9 +1,9 @@ # Implementation Readiness Status: -Ready for initial implementation after package installation and framework selection. +Real app scaffold, local-first import flow, analysis/reporting packages, CI gates, and hosted staging path are implemented. Remaining readiness work is MVP hardening and production verification. -What is already decided: +What is already decided and implemented: - Product name is Forage. - MVP uses GitHub App user authorization. - Repository data is stored locally in IndexedDB. @@ -14,40 +14,52 @@ What is already decided: - Base scoring is user-agnostic. - Personalized Match Score is deferred. - Categories start broad, then narrow after real data review. +- `apps/web` is an Astro app with Svelte islands. +- `apps/worker` is a Cloudflare Worker for auth/session/settings and GitHub API proxying. +- `packages/shared`, `packages/core`, `packages/github`, `packages/analysis`, and `packages/reporting` hold cross-boundary contracts and logic. +- Browser IndexedDB schema version 3 stores repositories, import events, local metadata, and analysis results. +- JSON and CSV exports are available from local browser data. +- Cloudflare Pages, Worker, Durable Object, KV, OpenTofu, deployment workflow, and hosted smoke check paths are documented and partially proven through staging. -Next implementation sequence: -1. Install project dependencies and lock package manager. -2. Scaffold the real Astro web app in `apps/web`. -3. Scaffold the Cloudflare Worker auth/session backend in `apps/worker`. -4. Move shared contracts from `packages/shared` into compiled TypeScript packages. -5. Move GitHub import logic from the pre-MVP server into `packages/github`. -6. Implement IndexedDB storage and migrations in the web app. -7. Implement category matching in `packages/analysis`. -8. Implement versioned foundational scoring in `packages/analysis`. -9. Implement JSON and CSV exports in `packages/reporting`. -10. Keep `apps/pre-mvp` until the real vertical slice covers the same behavior. +Current implementation sequence: +1. Harden import resilience: retry/recovery UX, cancellation, rate-limit display, and stalled page recovery. +2. Tighten first-run and re-auth UX across local, staging, and production origins. +3. Finish settings/privacy surface and make analytics opt-in state obvious. +4. Verify production GitHub App, Cloudflare secrets, Worker bindings, custom domains, and hosted smoke results. +5. Review real account imports to tune category weights, labels, score explanations, and edge cases. +6. Add broader browser smoke coverage after the primary app flows stabilize. +7. Keep `apps/pre-mvp` until the real app has fully replaced the same validation value in tests and docs. -First implementation milestone: -The real app should reproduce the pre-MVP flow with production-shaped boundaries: +Implemented first milestone: +The real app reproduces the pre-MVP flow with production-shaped boundaries: - GitHub App authorization - Import starred repositories - Store records in IndexedDB - Record import events - Run browser worker analysis -- Display a basic table -- Export JSON +- Display dashboard/table/search/filter/sort views +- Export JSON and CSV Definition of ready for broader MVP work: - The real app matches or exceeds the pre-MVP spike behavior. - Shared packages have tests around normalization, category matching, scoring, and export shape. - Auth/session behavior is documented and deployable to Cloudflare. - Local schema versions and migrations are implemented before any public testing. +- Hosted staging smoke checks pass after deploy. Definition of ready for hosted testing: -- Cloudflare Pages and Worker hostnames are finalized. +- Cloudflare Pages and Worker staging hostnames are finalized. - Worker `AUTH_COORDINATOR` Durable Object binding and migration are configured. - Worker `SETTINGS_KV` namespaces are created and bound. - GitHub token expiration behavior is reconnect-only for MVP. - Server-side settings, sessions, OAuth state, and token records have deletion behavior. - Production secrets and Cloudflare bindings are configured outside git. - Security header and CORS behavior is verified on HTTPS. + +Definition of ready for production launch: +- Production hostnames and GitHub App callback URLs are final. +- Production Worker secrets and Cloudflare bindings are configured outside git. +- Hosted smoke passes for production in public mode. +- Staging remains protected by Cloudflare Access. +- Traffic controls are active and do not break GitHub OAuth start/callback. +- Privacy checks confirm repository data remains browser-local. diff --git a/docs/17-ci-and-quality-gates.md b/docs/17-ci-and-quality-gates.md index 68b7160..56cb31d 100644 --- a/docs/17-ci-and-quality-gates.md +++ b/docs/17-ci-and-quality-gates.md @@ -47,7 +47,7 @@ Current Worker API contract coverage: Hosted smoke command: - Script: `scripts/check-hosted-smoke.mjs` -- Command: `FORAGE_WEB_ORIGIN=https://forage.example.com FORAGE_WORKER_ORIGIN=https://api.forage.example.com pnpm smoke:hosted` +- Command: `FORAGE_WEB_ORIGIN=https://forage.shrimpworks.dev FORAGE_WORKER_ORIGIN=https://api.forage.shrimpworks.dev pnpm smoke:hosted` - Purpose: verify deployed Worker health, Worker CORS/preflight, unauthenticated session shape, OAuth start redirect/PKCE setup, Pages headers, CSP Worker origin, and basic rendered app HTML - Staging can use `FORAGE_WEB_SMOKE_MODE=access-protected` when the web hostname is behind Cloudflare Access. - This is not part of the default CI gate because it requires live hosted domains. diff --git a/docs/20-hosting-and-security.md b/docs/20-hosting-and-security.md index 0dac9e1..54d61ff 100644 --- a/docs/20-hosting-and-security.md +++ b/docs/20-hosting-and-security.md @@ -10,8 +10,14 @@ Use two Cloudflare properties: - `apps/web`: Cloudflare Pages static Astro deployment. - `apps/worker`: Cloudflare Worker API for GitHub auth, session, settings, and GitHub API proxying. -The web app should be served from the product hostname, for example `https://forage.example.com`. -The Worker should be served from a separate API hostname, for example `https://api.forage.example.com`. +The web app should be served from the product hostname. +The Worker should be served from a separate API hostname. + +Current hostnames: +- Staging web: `https://forage-staging.shrimpworks.dev` +- Staging API: `https://api-staging.forage.shrimpworks.dev` +- Production web: `https://forage.shrimpworks.dev` +- Production API: `https://api.forage.shrimpworks.dev` Reasons: - The Astro app is static and local-first, so Pages is the simplest hosting boundary. @@ -44,12 +50,12 @@ Already implemented: - Initial OpenTofu scaffold for Cloudflare Pages, KV, domains, and environment output wiring. - OpenTofu-managed security controls for WAF geo challenges, API/auth rate limits, and optional staging Access allowlist. -Not production-ready yet: -- Production `SETTINGS_KV` namespace IDs are not configured in `wrangler.toml`. -- Pages preview deployment CORS behavior is not finalized. -- Production Cloudflare resource names and custom domains are not finalized. -- Production GitHub App callback URLs are still placeholders. -- OpenTofu has not been applied against the real Cloudflare account yet. +Remaining before production launch: +- Production GitHub App settings and callback URLs need final verification against the production Worker. +- Production Worker secrets and bindings need final verification after deploy. +- Production hosted smoke needs to pass in public mode. +- Pages preview access posture needs a final decision if preview URLs will be shared with testers. +- Staging Access, WAF, and rate-limit rules should be rechecked after any Cloudflare Access policy change. ## Production Storage Decision @@ -74,7 +80,7 @@ Use GitHub App user authorization through the web application flow. Production GitHub App settings: - Homepage URL: production web origin. -- Callback URL: production Worker callback URL, for example `https://api.forage.example.com/auth/github/callback`. +- Callback URL: production Worker callback URL, `https://api.forage.shrimpworks.dev/auth/github/callback`. - Add staging callback URL if a separate staging Worker is used. - Add local callback URL only for the development GitHub App, not the production app. - Repository permissions: none unless future features require installation access. @@ -102,14 +108,14 @@ Pages production config: - Build output directory: `apps/web/dist` - Production branch: `main` - Node version: `22` -- Environment variable: `PUBLIC_WORKER_ORIGIN=https://api.forage.example.com` +- Environment variable: `PUBLIC_WORKER_ORIGIN=https://api.forage.shrimpworks.dev` Worker production config: - Deploy command from `apps/worker`: `pnpm exec wrangler deploy --env production` - Non-secret vars: - `GITHUB_API_VERSION=2022-11-28` - - `GITHUB_REDIRECT_URI=https://api.forage.example.com/auth/github/callback` - - `WEB_ORIGIN=https://forage.example.com` + - `GITHUB_REDIRECT_URI=https://api.forage.shrimpworks.dev/auth/github/callback` + - `WEB_ORIGIN=https://forage.shrimpworks.dev` - Secrets: - `GITHUB_CLIENT_ID` - `GITHUB_CLIENT_SECRET` diff --git a/docs/21-hosting-ui-setup.md b/docs/21-hosting-ui-setup.md index 83a6fef..6783783 100644 --- a/docs/21-hosting-ui-setup.md +++ b/docs/21-hosting-ui-setup.md @@ -11,12 +11,15 @@ This document covers the manual UI configuration needed in GitHub and Cloudflare - Cloudflare KV stores only small settings/preferences records. - Repository data stays in the browser and is not stored in Cloudflare. -Replace the placeholder domains below before production: +Current Forage domains: -- Production web: `https://forage.example.com` -- Production API: `https://api.forage.example.com` -- Staging web: `https://forage-staging.example.com` -- Staging API: `https://api-staging.forage.example.com` +- Production web: `https://forage.shrimpworks.dev` +- Production API: `https://api.forage.shrimpworks.dev` +- Staging web: `https://forage-staging.shrimpworks.dev` +- Staging API: `https://api-staging.forage.shrimpworks.dev` +- Staging Pages branch URL: `https://staging.forage-web.pages.dev` + +If Forage moves to a different domain, update this document, OpenTofu variables, Pages variables, Worker variables, GitHub App callbacks, and hosted smoke environment variables together. ## GitHub App UI @@ -25,8 +28,8 @@ Create separate GitHub Apps for production and development/staging unless there Production GitHub App: - GitHub path: `Settings` -> `Developer settings` -> `GitHub Apps` -> `New GitHub App` - GitHub App name: `Forage` -- Homepage URL: production web URL, for example `https://forage.example.com` -- Callback URL: production Worker callback URL, for example `https://api.forage.example.com/auth/github/callback` +- Homepage URL: `https://forage.shrimpworks.dev` +- Callback URL: `https://api.forage.shrimpworks.dev/auth/github/callback` - Setup URL: leave blank for MVP - Webhook: disable active webhook unless a concrete webhook feature is added - Repository permissions: no repository permissions for MVP unless GitHub requires one for the stars endpoint @@ -37,9 +40,9 @@ Production GitHub App: Development or staging GitHub App: - App name: `Forage Dev` or `Forage Staging` -- Homepage URL: staging web URL or local web URL +- Homepage URL: `https://forage-staging.shrimpworks.dev` for staging, or local web URL for local-only testing - Callback URL for local development: `http://127.0.0.1:8787/auth/github/callback` -- Callback URL for staging: `https://api-staging.forage.example.com/auth/github/callback` +- Callback URL for staging: `https://api-staging.forage.shrimpworks.dev/auth/github/callback` - Use the same permission posture as production Values to copy from the GitHub App UI: @@ -61,14 +64,14 @@ Create the web app in Cloudflare Pages: - Node version: `22` Production environment variable: -- `PUBLIC_WORKER_ORIGIN=https://api.forage.example.com` +- `PUBLIC_WORKER_ORIGIN=https://api.forage.shrimpworks.dev` Staging or preview environment variable: -- `PUBLIC_WORKER_ORIGIN=https://api-staging.forage.example.com` +- `PUBLIC_WORKER_ORIGIN=https://api-staging.forage.shrimpworks.dev` Custom domains: -- Add the production web hostname, for example `forage.example.com` -- Add the staging web hostname if using a long-lived staging environment +- Add the production web hostname: `forage.shrimpworks.dev` +- Add the staging web hostname: `forage-staging.shrimpworks.dev` - Confirm HTTPS is active before relying on HSTS Important: @@ -82,20 +85,20 @@ The Worker can be deployed from CI or locally with Wrangler, but the Cloudflare Worker project: - Worker name: `forage-worker` -- Production route or custom domain: `https://api.forage.example.com` -- Staging route or custom domain: `https://api-staging.forage.example.com` +- Production route or custom domain: `https://api.forage.shrimpworks.dev` +- Staging route or custom domain: `https://api-staging.forage.shrimpworks.dev` Production Worker variables: - `ENVIRONMENT=production` - `GITHUB_API_VERSION=2022-11-28` -- `GITHUB_REDIRECT_URI=https://api.forage.example.com/auth/github/callback` -- `WEB_ORIGIN=https://forage.example.com` +- `GITHUB_REDIRECT_URI=https://api.forage.shrimpworks.dev/auth/github/callback` +- `WEB_ORIGIN=https://forage.shrimpworks.dev` Staging Worker variables: - `ENVIRONMENT=staging` - `GITHUB_API_VERSION=2022-11-28` -- `GITHUB_REDIRECT_URI=https://api-staging.forage.example.com/auth/github/callback` -- `WEB_ORIGIN=https://forage-staging.example.com` +- `GITHUB_REDIRECT_URI=https://api-staging.forage.shrimpworks.dev/auth/github/callback` +- `WEB_ORIGIN=https://forage-staging.shrimpworks.dev` Production Worker secrets: - `GITHUB_CLIENT_ID` @@ -158,10 +161,10 @@ If using the same token for the full OpenTofu root, include read/write permissio Use [Cloudflare Token Permissions](./24-cloudflare-token-permissions.md) for the full local infra, deploy workflow, and temporary recovery token permission matrix. Recommended staging posture: -- Protect `forage-staging.example.com` with Cloudflare Access. +- Protect `forage-staging.shrimpworks.dev` with Cloudflare Access. - Allow only explicit tester email addresses. - Keep Access cookies at `SameSite=Lax` so OAuth callback redirects can return to staging without requiring a hard refresh. -- Do not put `api-staging.forage.example.com` behind Access initially. +- Do not put `api-staging.forage.shrimpworks.dev` behind Access initially. - Use WAF and rate limiting for the API hostname so GitHub OAuth callbacks and credentialed CORS remain straightforward to test. Recommended production posture: @@ -205,8 +208,8 @@ Use GitHub environments named `staging` and `production`. Production should requ Run the automated hosted smoke check: ```sh -FORAGE_WEB_ORIGIN=https://forage.example.com \ -FORAGE_WORKER_ORIGIN=https://api.forage.example.com \ +FORAGE_WEB_ORIGIN=https://forage.shrimpworks.dev \ +FORAGE_WORKER_ORIGIN=https://api.forage.shrimpworks.dev \ pnpm smoke:hosted ``` @@ -223,11 +226,11 @@ pnpm smoke:hosted This script verifies Worker health, CORS, unauthenticated session shape, GitHub OAuth start redirect/PKCE setup, preflight headers, and either public Pages HTML/security headers or Cloudflare Access protection depending on `FORAGE_WEB_SMOKE_MODE`. Verify Worker health: -- `GET https://api.forage.example.com/api/health` +- `GET https://api.forage.shrimpworks.dev/api/health` - Expected: JSON with `ok: true` and `privacy_boundary: "no repository data stored server-side"` Verify Pages: -- Open `https://forage.example.com` +- Open `https://forage.shrimpworks.dev` - Confirm the app loads - Confirm browser devtools show CSP applied - Confirm no blocked app scripts from the expected Forage assets diff --git a/docs/23-deployment-automation.md b/docs/23-deployment-automation.md index 4834a8c..ce0f727 100644 --- a/docs/23-deployment-automation.md +++ b/docs/23-deployment-automation.md @@ -68,7 +68,7 @@ pnpm exec wrangler deploy --env staging Pages deploy: ```sh -PUBLIC_WORKER_ORIGIN=https://api-staging.forage.example.com pnpm --filter @forage/web build +PUBLIC_WORKER_ORIGIN=https://api-staging.forage.shrimpworks.dev pnpm --filter @forage/web build cd apps/worker pnpm exec wrangler pages deploy ../../apps/web/dist --project-name forage-web --branch staging ``` @@ -76,8 +76,8 @@ pnpm exec wrangler pages deploy ../../apps/web/dist --project-name forage-web -- Hosted smoke: ```sh -FORAGE_WEB_ORIGIN=https://staging.forage.example.com \ -FORAGE_WORKER_ORIGIN=https://api-staging.forage.example.com \ +FORAGE_WEB_ORIGIN=https://forage-staging.shrimpworks.dev \ +FORAGE_WORKER_ORIGIN=https://api-staging.forage.shrimpworks.dev \ FORAGE_SMOKE_EXPECT_PRODUCTION=false \ FORAGE_WEB_SMOKE_MODE=access-protected \ pnpm smoke:hosted @@ -85,7 +85,7 @@ pnpm smoke:hosted The deploy workflow sets `FORAGE_SMOKE_EXPECT_PRODUCTION=false` and `FORAGE_WEB_SMOKE_MODE=access-protected` automatically for staging because staging Worker config intentionally exposes non-secret setup diagnostics and the staging web hostname is behind Cloudflare Access. For production, the workflow sets `FORAGE_SMOKE_EXPECT_PRODUCTION=true` and `FORAGE_WEB_SMOKE_MODE=public` so the smoke check verifies public Pages HTML/security headers and confirms production config hides diagnostics. -When using Cloudflare Pages custom branch domains, the staging branch name should match the OpenTofu environment key. For example, the `staging` environment maps to `staging..pages.dev` and can be connected to a custom hostname such as `forage-staging.example.com`. +When using Cloudflare Pages custom branch domains, the staging branch name should match the OpenTofu environment key. For example, the `staging` environment maps to `staging..pages.dev` and can be connected to a custom hostname such as `forage-staging.shrimpworks.dev`. ## Quality Gates diff --git a/docs/README.md b/docs/README.md index 87f5054..ad75587 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,8 +1,27 @@ # Forage Documentation -This directory contains the planning and technical decision material for Forage. +This directory contains the product, architecture, implementation, hosting, and operations material for Forage. -## Product And Planning +## Start Here + +- Product context: [Executive Summary](./00-executive-summary.md) -> [Product Overview](./01-product-overview.md) +- System shape: [Architecture](./02-architecture.md) -> [Auth And Privacy](./11-auth-and-privacy.md) -> [Local Storage Schema](./13-storage-schema.md) +- MVP behavior: [MVP Acceptance Criteria](./14-mvp-acceptance-criteria.md) -> [Import Pipeline](./12-import-pipeline.md) -> [Dashboard And UX](./08-dashboard-and-ux.md) +- Analysis model: [Categories](./04-categories.md) -> [Scoring And Insights](./05-scoring-and-insights.md) -> [Analysis Review](./18-analysis-review.md) +- Hosted operations: [Hosting And Security Plan](./20-hosting-and-security.md) -> [Hosting UI Setup](./21-hosting-ui-setup.md) -> [Infrastructure As Code](./22-infrastructure-as-code.md) -> [Deployment Automation](./23-deployment-automation.md) -> [Cloudflare Token Permissions](./24-cloudflare-token-permissions.md) +- Current implementation state: [Implementation Readiness](./16-implementation-readiness.md) -> [Code Audit](./19-code-audit.md) -> [CI And Quality Gates](./17-ci-and-quality-gates.md) + +## Current Hosted Environments + +- Staging web: `https://forage-staging.shrimpworks.dev` +- Staging API: `https://api-staging.forage.shrimpworks.dev` +- Staging Pages branch URL: `https://staging.forage-web.pages.dev` +- Production web: `https://forage.shrimpworks.dev` +- Production API: `https://api.forage.shrimpworks.dev` + +Staging web is protected by Cloudflare Access. The staging API remains public for GitHub OAuth callbacks and credentialed CORS. + +## Product And Architecture - [Executive Summary](./00-executive-summary.md) - [Product Overview](./01-product-overview.md) @@ -16,7 +35,7 @@ This directory contains the planning and technical decision material for Forage. - [Roadmap](./09-roadmap.md) - [Open Questions](./10-open-questions.md) -## MVP Planning +## Implementation And Quality - [Auth And Privacy](./11-auth-and-privacy.md) - [Import Pipeline](./12-import-pipeline.md) @@ -27,6 +46,9 @@ This directory contains the planning and technical decision material for Forage. - [CI And Quality Gates](./17-ci-and-quality-gates.md) - [Analysis Review](./18-analysis-review.md) - [Code Audit](./19-code-audit.md) + +## Hosting And Operations + - [Hosting And Security Plan](./20-hosting-and-security.md) - [Hosting UI Setup](./21-hosting-ui-setup.md) - [Infrastructure As Code](./22-infrastructure-as-code.md)