diff --git a/AGENTS.md b/AGENTS.md index 30d73cd..89a40b1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -107,3 +107,13 @@ curl localhost:3000/api/admin/health # health check - **Notifications**: No alerting/notification hooks for provisioning failures or cost threshold breaches. + + + +## OpenWiki + +This repository uses OpenWiki for recurring code documentation. Start with `openwiki/quickstart.md`, then follow its links to architecture, workflows, domain concepts, operations, integrations, testing guidance, and source maps. + +The scheduled OpenWiki GitHub Actions workflow refreshes the repository wiki. Do not hand-edit generated OpenWiki pages unless explicitly asked; prefer updating source code/docs and letting OpenWiki regenerate. + + diff --git a/CLAUDE.md b/CLAUDE.md index 43c994c..106ecfa 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1 +1,11 @@ @AGENTS.md + + + +## OpenWiki + +This repository uses OpenWiki for recurring code documentation. Start with `openwiki/quickstart.md`, then follow its links to architecture, workflows, domain concepts, operations, integrations, testing guidance, and source maps. + +The scheduled OpenWiki GitHub Actions workflow refreshes the repository wiki. Do not hand-edit generated OpenWiki pages unless explicitly asked; prefer updating source code/docs and letting OpenWiki regenerate. + + diff --git a/openwiki/.last-update.json b/openwiki/.last-update.json new file mode 100644 index 0000000..edd9232 --- /dev/null +++ b/openwiki/.last-update.json @@ -0,0 +1,8 @@ +{ + "updatedAt": "2026-08-02T00:17:16.948Z", + "command": "update", + "gitHead": "9a9b5975cf43e502eab6759135b1298a8587eb09", + "model": "google/gemini-2.5-flash", + "status": "complete", + "language": "en" +} diff --git a/openwiki/INSTRUCTIONS.md b/openwiki/INSTRUCTIONS.md new file mode 100644 index 0000000..54a8cb3 --- /dev/null +++ b/openwiki/INSTRUCTIONS.md @@ -0,0 +1,8 @@ +# Wiki instructions — engineering mode + +Audience: engineers and coding agents working in this repository. + +Document the architecture, module boundaries, data flow, and non-obvious +design decisions. Prefer explaining *why* something is built the way it is +over restating what the code already makes obvious. Call out cross-repo or +cross-package dependencies explicitly. Keep pages scoped to one concept each. diff --git a/openwiki/backstage-portal.md b/openwiki/backstage-portal.md new file mode 100644 index 0000000..9a337bf --- /dev/null +++ b/openwiki/backstage-portal.md @@ -0,0 +1,60 @@ +--- +type: Application Documentation +title: Backstage Developer Portal +description: Documentation for the Backstage developer portal application. +tags: [application, backstage, developer-portal, frontend, typescript, yarn] +--- +# Backstage Developer Portal + +This directory (`backstage-portal/`) contains the Backstage developer portal application. Backstage is an open-source platform for building developer portals, providing a unified experience for managing services, tools, and documentation. + +## Purpose + +The Backstage portal serves as a central hub for developers within CrewCircle, offering a catalog of services, documentation, and various developer tools to streamline workflows and improve productivity. + +## Getting Started + +To start the Backstage application locally: + +```bash +yarn install +yarn start +``` + +This will typically launch the Backstage frontend and backend services. + +## Structure Overview + +``` +backstage-portal/ +├── app-config.yaml ← Main application configuration +├── app-config.production.yaml ← Production specific application configuration +├── backstage.json ← Backstage project metadata +├── catalog-info.yaml ← Example catalog entity definitions +├── examples/ ← Example entity and template definitions +├── packages/ ← Internal Backstage packages (e.g., app, backend) +│ ├── app/ ← Frontend application for Backstage UI +│ └── backend/ ← Backend services for Backstage +├── plugins/ ← Custom or external Backstage plugins +└── yarn.lock ← Yarn dependency lock file +``` + +## Key Features + +* **Software Catalog**: Discover and manage all software components, services, and APIs. +* **Scaffolding**: Create new projects and components from templates. +* **Documentation**: Centralized documentation for services and APIs. +* **Integrations**: Extensible with various plugins for integrating with other developer tools. + +## Configuration + +The primary configuration files are `app-config.yaml` and `app-config.production.yaml`. These files define various aspects of the Backstage application, including: + +* Backend service settings. +* Catalog entity providers. +* Authentication providers. +* Plugin configurations. + +## Development + +For local development, after cloning the repository, ensure all dependencies are installed with `yarn install` and then start the application with `yarn start`. The Backstage documentation provides comprehensive guides for extending and customizing the portal. diff --git a/openwiki/index.md b/openwiki/index.md new file mode 100644 index 0000000..2497b00 --- /dev/null +++ b/openwiki/index.md @@ -0,0 +1,15 @@ +--- +okf_version: "0.1" +--- + +# Files + +- [Backstage Developer Portal](backstage-portal.md) - Documentation for the Backstage developer portal application. +- [CrewCircle Monorepo Quickstart](quickstart.md) - A quickstart guide to the CrewCircle monorepo, outlining its key components and how to get started. +- [Sisyphus Content and Plans](sisyphus.md) - Documentation for the .sisyphus directory, containing strategic plans and content for various initiatives. +- [Next.js Website Application](website.md) - Documentation for the main Next.js web application, serving as the marketing site and user-facing interfaces. + +# Directories + +- [packages](packages/) +- [scripts](scripts/) diff --git a/openwiki/packages/account-setup.md b/openwiki/packages/account-setup.md new file mode 100644 index 0000000..2b57917 --- /dev/null +++ b/openwiki/packages/account-setup.md @@ -0,0 +1,64 @@ +--- +type: Package Documentation +title: Account Setup Package +description: Documentation for the account-setup package, responsible for automating vendor account creation. +tags: [package, account-setup, automation, playwright] +--- +# Account Setup Package + +The `@crewcircle/account-setup` package (`packages/account-setup`) is responsible for automating the one-time creation and configuration of various vendor accounts. It utilizes Playwright for browser automation to interact with vendor portals. + +## Purpose + +This package streamlines the initial setup process for new projects by programmatically creating accounts on services such as Cloudflare, GitHub, Stripe, and others, as defined in its `creators/` directory. + +## Structure + +``` +packages/account-setup/ +├── creators/ ← Modules for specific vendor account creation +│ ├── anthropic.py +│ ├── cloudflare.py +│ ├── crazy_domains.py +│ ├── dataforseo.py +│ ├── digitalocean.py +│ ├── doppler.py +│ ├── github.py +│ ├── google_cloud.py +│ ├── pulumi_account.py +│ ├── resend.py +│ ├── sentry.py +│ ├── stripe.py +│ ├── supabase.py +│ └── tally.py +├── lib/ ← Helper utilities (browser automation, Doppler integration, human interaction) +│ ├── browser.py +│ ├── doppler_store.py +│ └── human_pause.py +├── setup.py ← Main script to run account creation +├── config.py ← Configuration management +└── AGENT.md ← Agent instructions related to this package +``` + +## Usage + +To run the account setup process, navigate to the package directory and execute the `setup.py` script: + +```bash +cd packages/account-setup +uv run python setup.py +``` + +**Prerequisites:** + +* Python 3.11+ +* `uv` (or `pip`) +* Playwright browsers installed (e.g., `uv run playwright install chromium`) +* Environment variables configured in `.env.local` (e.g., `CC_EMAIL`, `CC_COMPANY`, `CC_GITHUB_USERNAME`). Refer to `packages/account-setup/.env.example` for required variables. + +## Key Functionality + +* **Automated Account Creation**: Scripts within `creators/` handle the specific logic for creating accounts on different platforms. +* **Browser Automation**: Leverages Playwright to simulate user interactions with web interfaces. +* **Doppler Integration**: The `lib/doppler_store.py` module suggests integration with Doppler for secure secret management. +* **Human Interaction**: `lib/human_pause.py` indicates points where manual intervention or verification might be required during the automated process. diff --git a/openwiki/packages/admin-ui.md b/openwiki/packages/admin-ui.md new file mode 100644 index 0000000..fca31d1 --- /dev/null +++ b/openwiki/packages/admin-ui.md @@ -0,0 +1,61 @@ +--- +type: Package Documentation +title: Admin UI Package +description: Documentation for the admin-ui package, containing shared UI components for CrewCircle internal administrative tools. +tags: [package, admin, ui, react, components, shared] +--- +# Admin UI Package + +The `@crewcircle/admin-ui` package (`packages/admin-ui`) provides a collection of shared UI components specifically designed for CrewCircle's internal administrative tools. This package is private, meaning its intended use is within the monorepo for building consistent and efficient admin interfaces. + +## Purpose + +The primary goal of this package is to establish a consistent look and feel and to reuse common UI patterns across different administrative applications. By centralizing these components, development effort is reduced, and the user experience for internal tools is improved. + +## Structure + +``` +packages/admin-ui/ +├── package.json ← npm package metadata +├── src/ ← Source code for UI components +│ ├── admin-shell.tsx ← Admin shell layout component +│ ├── admin-topbar.tsx ← Admin top bar component +│ ├── cost-chart.tsx ← Component for displaying cost charts +│ ├── data-table.tsx ← Generic data table component +│ ├── index.ts ← Barrel export for components +│ ├── project-card.tsx ← Component for displaying project information +│ ├── stat-card.tsx ← Component for displaying statistics +│ ├── status-badge.tsx ← Component for displaying status badges +│ └── types.ts ← TypeScript type definitions +└── tsconfig.json ← TypeScript configuration +``` + +## Key Functionality and Components + +* **Layout Components**: `admin-shell.tsx` and `admin-topbar.tsx` provide the fundamental layout and navigation for admin applications. +* **Data Visualization**: `cost-chart.tsx` is a specialized component for presenting cost-related data graphically. +* **Data Display**: `data-table.tsx` offers a reusable solution for presenting tabular data, while `project-card.tsx` and `stat-card.tsx` are designed for displaying specific types of information in a card format. +* **Status Indicators**: `status-badge.tsx` provides a visual cue for various statuses. + +## Usage + +Components from this package are consumed by other CrewCircle applications that require administrative interfaces, such as the `src/app/admin` pages in the main Next.js website. Developers would import and utilize these React components to build their admin UIs. + +```tsx +import { AdminShell, DataTable, StatCard } from "@crewcircle/admin-ui"; + +function MyAdminPage() { + return ( + + + {/* ... other admin content */} + + ); +} +``` + +## Technologies + +* **React**: For building user interfaces. +* **TypeScript**: Ensures type safety and improves developer experience. +* **Tailwind CSS Utilities**: `class-variance-authority`, `clsx`, and `tailwind-merge` indicate the use of utility-first CSS principles, likely with Tailwind CSS for styling. diff --git a/openwiki/packages/auth.md b/openwiki/packages/auth.md new file mode 100644 index 0000000..1ba0664 --- /dev/null +++ b/openwiki/packages/auth.md @@ -0,0 +1,81 @@ +--- +type: Package Documentation +title: Authentication Package +description: Documentation for the auth package, providing Supabase Auth and multi-tenant organization helpers. +tags: [package, auth, supabase, multi-tenancy, rls, typescript, python, fastapi] +--- +# Authentication Package + +The `@crewcircle/auth` package (`packages/auth`) provides authentication services and multi-tenant organization helpers for CrewCircle applications. It integrates with Supabase Auth and offers both TypeScript (for Next.js) and Python (for FastAPI) helpers, along with a shared SQL migration for the organization model. + +## Purpose + +This package centralizes authentication logic, user and organization management, and role-based access control (RBAC) across CrewCircle applications. It simplifies the implementation of secure, multi-tenant features. + +## Structure + +``` +packages/auth/ +├── package.json ← npm package metadata (TypeScript) +├── tsconfig.json ← TypeScript configuration +├── src/ ← TypeScript source for Next.js helpers +│ ├── types.ts ← TypeScript type definitions (Organization, Role, UserWithOrg, AuthContext) +│ ├── client.ts ← Next.js browser helpers (createAuthClient, signIn, etc.) +│ ├── server.ts ← Next.js server helpers (createServerClient, requireAuth, requireOrg) +│ ├── hooks.ts ← React hooks (useAuth, useOrg, useIsMember) +│ └── index.ts ← Barrel export for TypeScript modules +├── migrations/ ← SQL migrations for Supabase +│ └── 001_auth_helpers.sql ← Supabase SQL: org, members, roles, RLS, JWT helper +└── python/ ← Python source for FastAPI helpers + ├── pyproject.toml ← Python project metadata + └── crewcircle_auth/ ← Python package + ├── __init__.py ← FastAPI dependencies: require_auth, require_org + ├── client.py ← Supabase client factories (anon + service-role) + ├── models.py ← Pydantic models: Organization, Role, UserWithOrg + └── dependencies.py ← FastAPI Depends() helpers +``` + +## Setup + +### 1. Run the SQL Migration + +Apply `migrations/001_auth_helpers.sql` to your Supabase project. This script creates: + +* `public.organizations` table: Stores tenant organizations. +* `public.organization_members` table: Manages user membership within organizations. +* `public.roles` table: Defines RBAC roles (owner, admin, member, viewer). +* `public.set_current_org(uuid)` RPC function: Allows switching the active organization within a user's JWT. +* Row-Level Security (RLS) policies for all relevant tables. + +### 2. Environment Variables + +Configure the following environment variables, typically via Doppler: + +* `NEXT_PUBLIC_SUPABASE_URL`: Your Supabase project URL (for client-side/SSR). +* `NEXT_PUBLIC_SUPABASE_ANON_KEY`: Your Supabase public anonymous key (for client-side/SSR). +* `SUPABASE_SERVICE_ROLE_KEY`: Your Supabase service role key (server-only, bypasses RLS). **Never expose this on the client-side.** + +### 3. Installation + +* **Next.js (TypeScript)**: + ```bash + npm install @crewcircle/auth @supabase/supabase-js @supabase/ssr + ``` +* **FastAPI (Python)**: + ```bash + cd packages/auth/python + pip install -e . + ``` + +## Usage + +### TypeScript (Next.js) + +The package provides React hooks and server-side helpers for seamless integration with Next.js applications: + +* **Client Components**: Use `useAuth()` and `useOrg()` hooks to access user and organization context in browser components. +* **Server Components/Route Handlers**: Use `requireAuth()` and `requireOrg()` for server-side authentication and authorization checks. + +### Python (FastAPI) + +FastAPI applications can use `require_auth` and `require_org` dependencies to enforce authentication and organization context for API endpoints. diff --git a/openwiki/packages/database.md b/openwiki/packages/database.md new file mode 100644 index 0000000..6a1f783 --- /dev/null +++ b/openwiki/packages/database.md @@ -0,0 +1,84 @@ +--- +type: Package Documentation +title: Database Package +description: Documentation for the database package, handling Supabase Postgres client factories, RLS helpers, and base schema. +tags: [package, database, supabase, postgres, rls, typescript, python] +--- +# Database Package + +The `@crewcircle/database` package (`packages/database`) provides Supabase Postgres client factories, Row-Level Security (RLS) helpers, and the base schema for multi-tenant CrewCircle applications. It supports both TypeScript (for Next.js) and Python (for FastAPI). + +## Purpose + +This package standardizes database access and ensures data isolation for multi-tenant applications by providing utilities for creating database clients, managing RLS policies, and defining the core database schema. + +## Structure + +``` +packages/database/ +├── package.json ← npm package metadata (TypeScript) +├── tsconfig.json ← TypeScript configuration +├── src/ ← TypeScript source +│ ├── client.ts ← Supabase client factories (createDatabaseClient, createServiceClient, createOrgClient) +│ ├── index.ts ← Barrel export +│ ├── rls.ts ← RLS policy generation utilities (setupTableRLS) +│ └── schema.ts ← TypeScript type definitions for the database schema +├── migrations/ ← SQL migrations for Supabase +│ ├── 001_base_schema.sql ← Initial base schema +│ ├── 002_app_schema_template.sql ← Application schema template +│ ├── 003_llm_usage_logs.sql ← LLM usage logging schema +│ ├── 004_provisioning_jobs.sql ← Provisioning jobs schema +│ └── 005_fixed_costs.sql ← Fixed costs schema +└── python/ ← Python source + ├── pyproject.toml ← Python project metadata + └── crewcircle_database/ ← Python package + ├── __init__.py ← Imports for Python utilities + ├── client.py ← Supabase client factories (create_database_client, OrgContext) + ├── migrate.py ← Database migration scripts + ├── rls.py ← Python RLS utilities (setup_table_rls) + └── models.py (implied) ← Pydantic models for database entities +``` + +## Setup + +### Environment Variables + +Database access relies on the following environment variables, typically managed via Doppler: + +* `SUPABASE_URL`: The URL of the Supabase project. +* `SUPABASE_ANON_KEY`: The public anonymous key for Supabase (used by browser/SSR clients). +* `SUPABASE_SERVICE_ROLE_KEY`: The service role key for Supabase (server-only, bypasses RLS). + +### Installation + +* **TypeScript / Next.js**: The package is consumed directly as TypeScript source. `npm install` from the workspace root installs dependencies. +* **Python / FastAPI**: Install the Python package in editable mode: + ```bash + cd packages/database/python + pip install -e . + ``` + +## Usage + +### Creating Clients + +The package provides functions to create different types of Supabase clients: + +* `createDatabaseClient()`: Anonymous client, subject to user-session RLS. +* `createServiceClient()`: Service-role client (server only), which bypasses RLS. +* `createOrgClient('org_abc123')`: Client scoped to a specific organization. + +### Row-Level Security (RLS) + +RLS is crucial for multi-tenancy, isolating data by `org_id`. Each request must set the `org_id`. + +* **TypeScript**: Use `supabase.rpc('set_org_id', { p_org_id: '...' })` to set the organization ID before querying. +* **Python**: Use the `OrgContext` context manager with `psycopg` to automatically set the `org_id` for database operations within the context. + +### Generating RLS Policies + +For new tables with an `org_id` column, the `setupTableRLS` (TypeScript) and `setup_table_rls` (Python) utilities can generate the necessary `ALTER TABLE` and `CREATE POLICY` statements. + +### Database Types + +TypeScript applications can import `Database` types from `@crewcircle/database/schema` to get strong typing for Supabase client queries. diff --git a/openwiki/packages/docs-gen.md b/openwiki/packages/docs-gen.md new file mode 100644 index 0000000..81f3e9b --- /dev/null +++ b/openwiki/packages/docs-gen.md @@ -0,0 +1,47 @@ +--- +type: Package Documentation +title: Docs Generator Package +description: Documentation for the docs-gen package, a utility for generating documentation from templates. +tags: [package, docs, generation, templates] +--- +# Docs Generator Package + +The `@crewcircle/docs-gen` package (`packages/docs-gen`) is a utility designed to generate documentation from templates. It appears to support different documentation formats or audiences, as suggested by its template files. + +## Purpose + +This package automates the process of creating various documentation artifacts, likely using predefined templates and potentially injecting dynamic content. + +## Structure + +``` +packages/docs-gen/ +├── CHANGELOG.md ← Change log for the package +├── package.json ← npm package metadata +├── src/ ← Source code for the documentation generator +│ ├── cli.ts ← Command-line interface for the generator +│ ├── defaults.ts ← Default configurations or content +│ └── index.ts ← Main entry point for the generator logic +├── templates/ ← Documentation templates +│ ├── INSTRUCTIONS.client.md ← Template for client-facing instructions +│ └── INSTRUCTIONS.engineering.md ← Template for engineering instructions +├── test/ ← Unit tests for the package +└── tsconfig.json ← TypeScript configuration +``` + +## Usage + +Based on the `cli.ts` file, this package likely provides a command-line interface for generating documentation. Specific commands and options would be detailed within the CLI or its internal documentation. + +Example (speculative): + +```bash +npm run docs-gen generate --template client --output ./docs/client-instructions.md +``` + +## Key Functionality + +* **Template-based Generation**: Uses predefined Markdown templates to structure generated documents. +* **Command-Line Interface**: Provides a CLI for easy execution and integration into build processes. +* **Audience-specific Templates**: Supports different templates for various audiences (e.g., client, engineering) to tailor the content and tone. + diff --git a/openwiki/packages/index.md b/openwiki/packages/index.md new file mode 100644 index 0000000..ebffb7a --- /dev/null +++ b/openwiki/packages/index.md @@ -0,0 +1,10 @@ +# Files + +- [Account Setup Package](account-setup.md) - Documentation for the account-setup package, responsible for automating vendor account creation. +- [Admin UI Package](admin-ui.md) - Documentation for the admin-ui package, containing shared UI components for CrewCircle internal administrative tools. +- [Authentication Package](auth.md) - Documentation for the auth package, providing Supabase Auth and multi-tenant organization helpers. +- [Database Package](database.md) - Documentation for the database package, handling Supabase Postgres client factories, RLS helpers, and base schema. +- [Docs Generator Package](docs-gen.md) - Documentation for the docs-gen package, a utility for generating documentation from templates. +- [Infrastructure Package](infra.md) - Documentation for the infra package, managing CrewCircle's cloud infrastructure with Pulumi. +- [Knowledge Layer Package](knowledge.md) - Documentation for the knowledge package, providing an internal organization memory and external application context layer. +- [Observability Package](observability.md) - Documentation for the observability package, providing tools for monitoring, logging, and performance tracking. diff --git a/openwiki/packages/infra.md b/openwiki/packages/infra.md new file mode 100644 index 0000000..70b1c92 --- /dev/null +++ b/openwiki/packages/infra.md @@ -0,0 +1,109 @@ +--- +type: Package Documentation +title: Infrastructure Package +description: Documentation for the infra package, managing CrewCircle's cloud infrastructure with Pulumi. +tags: [package, infrastructure, pulumi, iac, esc, doppler, github, sentry, cloudflare] +--- +# Infrastructure Package + +The `@crewcircle/infra` package (`packages/infra`) manages CrewCircle's cloud infrastructure using Pulumi (Python) with Pulumi ESC for secrets management. + +## Architecture + +This package defines and manages infrastructure resources through two main Pulumi stacks: + +1. **Master Stack**: Manages organization-level resources that are shared across all projects. This stack is typically deployed once. +2. **Per-Project Stack**: Manages resources specific to individual projects, deployed whenever a new project is provisioned. + +### Directory Structure + +``` +packages/infra/ +├── __main__.py ← Master stack entry point (org-level resources) +├── Pulumi.yaml ← Project definition for the master stack +├── Pulumi.master.yaml ← Master stack configuration (ESC binding) +├── esc/ ← Pulumi ESC environment definitions +│ ├── master.yaml ← ESC environment for the master stack (imports Doppler) +│ └── project.yaml ← ESC environment template for per-project stacks +├── shared/ ← Shared Python modules for infrastructure logic +│ ├── config.py ← Dotenv loader + Config dataclass +│ └── registry.py ← Manages `registry.json` for project lifecycle +├── bin/ ← Helper scripts for managing projects +│ ├── newproject ← Script to create a new project stack, ESC environment, and deploy +│ └── killproject ← Script to destroy a project stack and its ESC environment +├── template/ ← Template for new project stacks +│ ├── Pulumi.yaml ← Template project definition +│ ├── __main__.py ← Template stack entry point (per-project resources) +│ └── requirements.txt ← Template Python dependencies +├── src/crewcircle_infra/ ← Python source for infrastructure utilities +├── requirements.txt ← Master Python dependencies +└── pyproject.toml +``` + +### Two Stacks + +| Stack | Entry Point | Purpose | +| :---------------- | :---------------- | :--------------------------------------------------------- | +| **Master** | `__main__.py` | Configures GitHub organization secrets, Sentry team (run once) | +| **Per-Project** | `template/__main__.py` | Provisions Supabase, Stripe, GitHub repo, Cloudflare DNS, Doppler, Sentry project for each new project | + +### ESC Environment Import Chain + +Pulumi ESC is used for secrets management, with a clear import hierarchy: + + +```text +graph TD + A[Doppler crewcircle-master/prod] --> B[crewcircle/master]; + B -- imports Doppler, maps to pulumiConfig --> C[crewcircle/]; + C -- imports master, adds per-project config --> D[Pulumi Stack] +``` + +This chain ensures that sensitive credentials are sourced from Doppler and securely provided to the Pulumi stacks. + +## Deployment Tasks + +### Deploy Master Stack + +This task is performed during initial setup or after updates to master stack resources. + +```bash +cd packages/infra + +# Install master Python dependencies +pip install -r requirements.txt + +# Create / select master stack +pulumi stack init master 2>/dev/null || pulumi stack select master + +# Bind ESC environment +pulumi config env add crewcircle/master --yes + +# Preview and Deploy +pulumi preview +pulumi up --yes +``` + +**What it creates:** + +* GitHub Actions Secrets: `DOPPLER_TOKEN`, `SUPABASE_ACCESS_TOKEN`, `SENTRY_AUTH_TOKEN`, `CLOUDFLARE_API_TOKEN` +* GitHub Actions Variable: `ESC_ENV=crewcircle/master` +* Sentry team: `core` + +**Prerequisites:** + +* Doppler project `crewcircle-master/prod` with all secrets populated. +* ESC environment `crewcircle/master` created and configured to point at the Doppler project. +* `pulumi` CLI installed and logged in. +* Pulumi access token available via `PULUMI_ACCESS_TOKEN` environment variable. + +### Provision a New Project + +New projects are provisioned using the `newproject` helper script: + +```bash +cd packages/infra +./bin/newproject "" "" [price-cents] +``` + +This script automates the creation of a dedicated Pulumi stack and ESC environment for the new project, along with deploying its associated resources. diff --git a/openwiki/packages/knowledge.md b/openwiki/packages/knowledge.md new file mode 100644 index 0000000..6a11c10 --- /dev/null +++ b/openwiki/packages/knowledge.md @@ -0,0 +1,49 @@ +--- +type: Package Documentation +title: Knowledge Layer Package +description: Documentation for the knowledge package, providing an internal organization memory and external application context layer. +tags: [package, knowledge-graph, rag, ai-memory, vector-database, postgres] +--- +# Knowledge Layer Package + +The `@crewcircle/knowledge` package (`packages/knowledge`) serves as CrewCircle's internal organization memory and external application context layer. It is designed to support AI functionalities, including Retrieval Augmented Generation (RAG). + +## Purpose + +This package aims to create a centralized, queryable knowledge base by ingesting data from various sources. It facilitates enhanced AI capabilities by providing relevant context to large language models. + +## Structure + +``` +packages/knowledge/ +├── package.json ← npm package metadata +├── src/ ← Source code +│ ├── backup/ ← R2 backup utilities (e.g., r2-backup.ts) +│ ├── core/ ← Core knowledge graph logic (e.g., knowledge-graph.ts, types.ts) +│ ├── external/ ← Integrations with external knowledge sources +│ ├── internal/ ← Logic for internal knowledge sources +│ └── providers/ ← Data storage and retrieval providers +│ ├── embedded/ ← Embedded data providers +│ ├── mock/ ← Mock provider for testing +│ └── postgres/ ← Postgres provider (likely utilizing pgvector) +├── test/ ← Unit and integration tests +├── tsconfig.json ← TypeScript configuration +└── vitest.config.ts ← Vitest test configuration +``` + +## Key Functionality + +* **Knowledge Graph**: Central component for organizing and querying knowledge. +* **RAG (Retrieval Augmented Generation)**: Provides context to AI models for generating more informed responses. +* **Data Ingestion**: Supports ingestion from various internal and external sources. +* **Vector Database Integration**: Likely uses `pgvector` with PostgreSQL for efficient semantic search and retrieval. +* **Backup**: Includes utilities for backing up knowledge data, e.g., to Cloudflare R2 storage. +* **Providers**: Modular design allowing different storage and retrieval backends (e.g., PostgreSQL, mock). + +## Technologies Used + +* **TypeScript**: Primary development language. +* **PostgreSQL with `pgvector`**: For storing and querying vector embeddings. +* **OpenAI/AI SDK**: For interacting with large language models. +* **Cloudflare R2**: For object storage and backups. + diff --git a/openwiki/packages/observability.md b/openwiki/packages/observability.md new file mode 100644 index 0000000..2073f5f --- /dev/null +++ b/openwiki/packages/observability.md @@ -0,0 +1,44 @@ +--- +type: Package Documentation +title: Observability Package +description: Documentation for the observability package, providing tools for monitoring, logging, and performance tracking. +tags: [package, observability, monitoring, logging, metrics, llm-cost, sentry, uptime] +--- +# Observability Package + +The `@crewcircle/observability` package (`packages/observability`) provides a suite of tools and utilities for monitoring, logging, and tracking the health and performance of CrewCircle applications. It includes functionalities for LLM cost tracking, health checks, metrics collection, Sentry integration, and uptime monitoring. + +## Purpose + +This package centralizes observability concerns, enabling developers to gain insights into application behavior, identify issues, track resource usage, and ensure system reliability. + +## Structure + +``` +packages/observability/ +├── README.md ← Package documentation (this file) +├── llm_cost/ ← LLM cost tracking utilities +│ └── tracker.py ← Python script for tracking LLM costs +├── nextjs/ ← Next.js specific observability integrations +│ └── sentry.ts ← Sentry integration for Next.js applications +├── python/ ← Python utilities for observability +│ ├── crewcircle_observability/ +│ │ ├── __init__.py +│ │ ├── health.py ← Health check utilities +│ │ ├── logging.py ← Logging configurations and helpers +│ │ ├── metrics.py ← Metrics collection and reporting +│ │ └── sentry.py ← Sentry integration for Python applications +│ └── pyproject.toml +├── uptime/ ← Uptime monitoring utilities +│ └── check.py ← Python script for uptime checks +└── .gitignore +``` + +## Key Functionality + +* **LLM Cost Tracking**: The `llm_cost/tracker.py` module helps in monitoring and tracking the costs associated with Large Language Model (LLM) usage. +* **Health Checks**: Provides utilities (`python/crewcircle_observability/health.py`) to implement and report on application health, crucial for readiness and liveness probes in deployment environments. +* **Logging**: Offers centralized logging configurations and helpers (`python/crewcircle_observability/logging.py`) to ensure consistent and effective log collection. +* **Metrics**: Facilitates the collection and reporting of application metrics (`python/crewcircle_observability/metrics.py`) for performance monitoring and alerting. +* **Sentry Integration**: Integrates with Sentry for error tracking and performance monitoring. Dedicated modules for Next.js (`nextjs/sentry.ts`) and Python (`python/crewcircle_observability/sentry.py`) applications. +* **Uptime Monitoring**: Includes scripts (`uptime/check.py`) for performing uptime checks, potentially integrating with external monitoring services like Uptime Robot. diff --git a/openwiki/quickstart.md b/openwiki/quickstart.md new file mode 100644 index 0000000..b6c003f --- /dev/null +++ b/openwiki/quickstart.md @@ -0,0 +1,33 @@ +--- +type: Overview +title: CrewCircle Monorepo Quickstart +description: A quickstart guide to the CrewCircle monorepo, outlining its key components and how to get started. +tags: [quickstart, monorepo, overview] +--- +# CrewCircle Monorepo Quickstart + +Welcome to the CrewCircle monorepo! This repository contains various services, applications, and infrastructure code designed to support CrewCircle's operations. This document provides a high-level overview of the monorepo's structure and guides you to more detailed documentation for each major component. + +## Getting Started + +To get a local development environment up and running, refer to the root [README.md](/README.md) for initial setup instructions, including prerequisites for Python, Node.js, Playwright, and Pulumi. + +## Key Components + +The monorepo is organized into several key packages and applications: + +* **[Account Setup](/openwiki/packages/account-setup.md)**: Tools for automating the creation and configuration of various vendor accounts. +* **[Authentication](/openwiki/packages/auth.md)**: Centralized authentication services using Supabase, providing multi-tenant organization helpers for both TypeScript and Python applications. +* **[Database](/openwiki/packages/database.md)**: Database client factories, Row-Level Security (RLS) helpers, and the base schema for multi-tenant applications using Supabase Postgres. +* **[Docs Generator](/openwiki/packages/docs-gen.md)**: A utility for generating documentation within the monorepo. +* **[Infrastructure](/openwiki/packages/infra.md)**: Pulumi-based Infrastructure as Code (IaC) for managing CrewCircle's cloud resources, including master and per-project stacks. +* **[Knowledge Layer](/openwiki/packages/knowledge.md)**: An internal knowledge graph and external application context layer for AI memory and RAG (Retrieval Augmented Generation). +* **[Observability](/openwiki/packages/observability.md)**: A suite of tools for monitoring, logging, and tracking the health and performance of applications, including LLM cost tracking, health checks, metrics, Sentry integration, and uptime monitoring. +* **[Backstage Portal](/openwiki/backstage-portal.md)**: A developer portal built on Backstage, providing a centralized view for services, documentation, and tools. +* **[Website (Next.js)](/openwiki/website.md)**: The main Next.js web application, serving as the marketing site and potentially other user-facing interfaces. + +--- + +## Backlog + +* Explore `src/lib/admin` for potential admin-related documentation. diff --git a/openwiki/scripts/index.md b/openwiki/scripts/index.md new file mode 100644 index 0000000..80783d8 --- /dev/null +++ b/openwiki/scripts/index.md @@ -0,0 +1,3 @@ +# Files + +- [Social Media Automation Scripts](social.md) - Documentation for the scripts/social directory, which provides a Python toolkit for automating social media publishing. diff --git a/openwiki/scripts/social.md b/openwiki/scripts/social.md new file mode 100644 index 0000000..b5492ad --- /dev/null +++ b/openwiki/scripts/social.md @@ -0,0 +1,58 @@ +--- +type: Script Documentation +title: Social Media Automation Scripts +description: Documentation for the scripts/social directory, which provides a Python toolkit for automating social media publishing. +tags: [script, automation, social-media, python, substack, linkedin, x, youtube, scheduling] +--- +# Social Media Automation Scripts + +The `scripts/social/` directory contains a Python toolkit designed for automating the publishing of content to various social media platforms, including Substack, LinkedIn, and X (formerly Twitter). It allows for local scheduling of posts based on a YAML content calendar, eliminating the need for third-party scheduling tools. + +## Purpose + +This toolkit streamlines the process of distributing content across multiple social media channels, enabling consistent and timely communication with an audience without relying on external services that might incur additional costs or introduce vendor lock-in. + +## Structure + +``` +scripts/social/ +├── publish.py ← Main orchestrator script for publishing content +├── platforms/ ← Modules for interacting with specific social media platforms +│ ├── substack.py ← Handles publishing to Substack (via email-to-post) +│ ├── linkedin.py ← Integrates with LinkedIn API v2 +│ ├── twitter.py ← Integrates with X (Twitter) API v2 +│ └── youtube.py ← Placeholder for YouTube upload functionality +├── config.yaml.example ← Template for configuration settings (API keys, etc.) +├── calendar.yaml.example ← Template for the content publishing schedule +├── scheduler/ ← Example configurations for local scheduling tools +│ ├── crontab.txt ← Example cron job configuration +│ └── com.crewcircle.social.publish.plist ← Example `launchd` configuration for macOS +└── secret/ ← Directory for sensitive configuration and state (gitignored) + ├── config.yaml ← Your actual credentials and configuration + └── state.yaml ← Tracks previously published posts +``` + +## Setup + +1. **Copy Configuration Examples**: Copy `config.yaml.example` to `secret/config.yaml` and `calendar.yaml.example` to `calendar.yaml`. Fill in your specific credentials and content schedule. +2. **Install Dependencies**: The script requires `requests`, `requests-oauthlib`, and `pyyaml`. Install them in a virtual environment: + ```bash + python3 -m venv scripts/social/.venv + ./scripts/social/.venv/bin/pip install requests requests-oauthlib pyyaml + ``` +3. **Secrets Management**: Choose between Doppler (recommended for CrewCircle) or environment variables/`.env` files for managing API keys and sensitive information. Configure `secret/config.yaml` accordingly. +4. **Dry-Run Test**: Verify your setup by running the `publish.py` script in dry-run mode: + ```bash + ./scripts/social/.venv/bin/python scripts/social/publish.py --dry-run + ``` +5. **Schedule**: Set up recurring execution using `cron` (for Linux/macOS) or `launchd` (for macOS) as per the examples in `scripts/social/scheduler/`. + +## Platform-Specific Setup + +* **Substack**: Requires an email-to-post address from your publication settings and an app-specific SMTP password. +* **LinkedIn**: Involves creating a LinkedIn Developer application, requesting specific products (`Share on LinkedIn`, `Sign In with LinkedIn using OpenID Connect`), obtaining an OAuth token, and identifying person/company URNs. +* **X (Twitter)**: Notes the limitations of the free tier and the likelihood of needing a paid plan for write access. + +## Integration with `.sisyphus` + +The content and plans defined within the `.sisyphus/` directory, particularly `social-content-plan.md` and content assets under `content/phase-1/`, are directly consumed by these automation scripts. This ensures that the publishing process aligns with the broader content strategy. diff --git a/openwiki/sisyphus.md b/openwiki/sisyphus.md new file mode 100644 index 0000000..4a80713 --- /dev/null +++ b/openwiki/sisyphus.md @@ -0,0 +1,59 @@ +--- +type: Project Documentation +title: Sisyphus Content and Plans +description: Documentation for the .sisyphus directory, containing strategic plans and content for various initiatives. +tags: [project-management, content-strategy, planning, sisyphus] +--- +# Sisyphus: Content and Plans + +The `.sisyphus/` directory serves as a central repository for strategic plans and content-related assets across various project initiatives within CrewCircle. It organizes detailed documentation for implementation, infrastructure, and social media content, often structured by project phases. + +## Purpose + +This directory provides a structured approach to project planning and content creation, ensuring that strategic decisions, implementation details, and communication strategies are well-documented and easily accessible. + +## Structure + +``` +.sisyphus/ +├── content/ ← Content assets, organized by phases or campaigns +│ └── phase-1/ ← Content specific to Phase 1 (e.g., blog posts, social media updates) +│ ├── content-calendar.md +│ ├── launch-posts.md +│ ├── linkedin-week-1.md +│ ├── substack-article.md +│ ├── twitter-week-1.md +│ └── youtube-script.md +└── plans/ ← Strategic and implementation plans + ├── implementation-plan.md ← General implementation plan + ├── infra-module-implementation.md ← Detailed plan for infrastructure module implementation + ├── infrastructure-migration-plan.md ← Plan for infrastructure migration + ├── social-content-plan.md ← Plan for social media content strategy + └── spec-infra-module.md ← Specification for an infrastructure module +``` + +## Key Areas + +### Content + +The `content/` subdirectory, further organized by phases (e.g., `phase-1`), holds various content assets. This includes: + +* **Content Calendars**: Outlining publishing schedules. +* **Launch Posts**: Drafts and plans for product launches. +* **Social Media Content**: Specific posts and strategies for platforms like LinkedIn, Twitter (X). +* **Long-form Content**: Articles for platforms like Substack, and scripts for YouTube videos. + +This structure ensures a cohesive approach to content delivery and campaign management. + +### Plans + +The `plans/` subdirectory contains critical strategic and implementation documents. These documents provide in-depth details on: + +* **Implementation Strategies**: General approaches to project execution. +* **Infrastructure Design and Migration**: Detailed plans for evolving the technical infrastructure. +* **Social Content Strategy**: The overarching plan for CrewCircle's presence on social media platforms. +* **Technical Specifications**: Detailed specifications for key modules, such as infrastructure components. + +## Relation to other parts of the monorepo + +Content planned and documented within `.sisyphus/content` often correlates with the output of `scripts/social`, which automates publishing to various social media platforms. Similarly, plans under `.sisyphus/plans` directly influence the development and deployment managed by the `packages/infra` and other technical packages. diff --git a/openwiki/website.md b/openwiki/website.md new file mode 100644 index 0000000..aaf46de --- /dev/null +++ b/openwiki/website.md @@ -0,0 +1,66 @@ +--- +type: Application Documentation +title: Next.js Website Application +description: Documentation for the main Next.js web application, serving as the marketing site and user-facing interfaces. +tags: [application, nextjs, react, frontend, website, marketing] +--- +# Next.js Website Application + +This directory (`src/app/`) contains the main Next.js web application. It serves as the primary marketing website for CrewCircle and hosts various user-facing interfaces and content. + +## Purpose + +This application is built with Next.js to provide a fast, SEO-friendly, and scalable platform for CrewCircle's online presence, marketing efforts, and potentially core product features. + +## Getting Started + +To run the development server: + +```bash +npm run dev +# or +yarn dev +# or +pnpm dev +# or +bun dev +``` + +Open [http://localhost:3000](http://localhost:3000) with your browser to see the result. You can edit pages in `app/` and the changes will hot-reload. + +## Structure Overview + +The `src/app` directory follows Next.js's App Router conventions: + +``` +src/app/ +├── admin/ ← Admin-related pages and APIs +├── api/ ← API routes (e.g., admin, registry) +├── blog/ ← Blog posts +├── cardsnap/ ← CardSnap specific pages (e.g., privacy-policy) +├── documentation/ ← General documentation page +├── locations/ ← Location-specific pages +├── pricing/ ← Pricing page +├── privacy/ ← Privacy policy page +├── solutions/ ← Solutions overview page +├── terms/ ← Terms and conditions page +├── globals.css ← Global styles +├── layout.tsx ← Root layout for the application +├── page.tsx ← Main landing page +├── favicon.ico ← Favicon +├── robots.ts ← Robots.txt generation +└── sitemap.ts ← Sitemap generation +``` + +## Key Technologies + +* **Next.js**: React framework for building server-rendered and statically generated web applications. +* **React**: JavaScript library for building user interfaces. +* **TypeScript**: Statically typed superset of JavaScript. +* **Tailwind CSS (implied)**: Often used with Next.js for utility-first styling (indicated by `globals.css` and `postcss.config.mjs`). + +## Deployment + +This Next.js application is designed for deployment on platforms like Vercel, which provides seamless integration and optimization for Next.js projects. + +For more details on Next.js, refer to the [Next.js Documentation](https://nextjs.org/docs).