diff --git a/.claude/skills/initial-setup/SKILL.md b/.claude/skills/initial-setup/SKILL.md new file mode 100644 index 0000000..aca4e7b --- /dev/null +++ b/.claude/skills/initial-setup/SKILL.md @@ -0,0 +1,282 @@ +--- +name: initial-setup +description: First-time deployment wizard for databricks-claw — sets up Slack app, UC connection, secrets, and deploys +user-invocable: true +allowed-tools: Bash, Read, Edit, Write, Glob, Grep +--- + +# Initial Setup Wizard for databricks-claw + +Guide the user through a complete first-time deployment of their personal Claw instance. +Work through each phase IN ORDER. After each phase, confirm everything passed before moving on. +If something is missing or broken, STOP and help the user fix it before proceeding. + +CRITICAL: NEVER ask the user to paste or type a token, secret, or credential into this conversation. +Always use `databricks secrets` commands which accept input interactively (never echoed). + +--- + +## Phase 1 — Prerequisites Check + +Verify all required tools are installed and authenticated. + +1. Check `databricks` CLI is installed: + ``` + databricks --version + ``` +2. Check the CLI is authenticated and can reach the workspace: + ``` + databricks auth describe + ``` + Confirm the output shows a valid host and authentication method. Note the workspace host URL — you will need it later. +3. Check `uv` is installed: + ``` + uv --version + ``` +4. Verify the user is in the repo root by checking for `databricks.yml`: + ``` + test -f databricks.yml && echo "OK: in repo root" || echo "ERROR: databricks.yml not found" + ``` + +If any check fails, tell the user what to install/configure and STOP. + +--- + +## Phase 2 — Databricks App Configuration (app.yaml) + +### 2a. Check if app.yaml exists + +If `app.yaml` does NOT exist, copy from the template: +``` +cp app.example.yml app.yaml +``` + +### 2b. Walk through required env vars + +Read `app.yaml` and guide the user through each field. The required env vars are: + +| Env Var | Source | Description | +|---|---|---| +| `DATABRICKS_USER_PAT` | `valueFrom: user-pat` (secret) | User's Personal Access Token — injected from the `user-pat` secret resource in `resources/app.yml` | +| `ANTHROPIC_AUTH_TOKEN` | `valueFrom: user-pat` (same secret) | Passed to Claude Agent SDK — uses the same PAT | +| `ANTHROPIC_BASE_URL` | value | AI Gateway URL. Format: `https://.ai-gateway.cloud.databricks.com/anthropic` | +| `ANTHROPIC_MODEL` | value | AI Gateway serving endpoint name (default: `databricks-claude-sonnet-4-6`) | +| `ANTHROPIC_CUSTOM_HEADERS` | value | Must be `x-databricks-use-coding-agent-mode: true` | +| `CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS` | value | Must be `1` | +| `SLACK_UC_CONNECTION` | value | UC connection name for Slack Web API (e.g. `slack`) | +| `CLAW_MEMORY_VOLUME_PATH` | value | UC Volume path for memory. Format: `/Volumes//default/claw_memory` | + +Optional env vars (can be added later): +| Env Var | Description | +|---|---| +| `UC_MCP_CONNECTIONS` | Comma-separated UC connection names for external MCP servers | +| `UC_MANAGED_MCPS` | Comma-separated managed MCP specs (`genie:`, `vector-search:`, `sql:`, `uc-functions:`) | +| `UC_CUSTOM_MCPS` | Comma-separated `name=url` pairs for custom HTTP MCP servers | +| `CLAW_NAME` | Bot display name (default: `claw`) | +| `POLL_INTERVAL_SECONDS` | Polling interval in seconds (default: `5`) | +| `ACTIVE_THREAD_WINDOW_DAYS` | Days before threads are auto-evicted (default: `7`) | +| `MAX_ACTIVE_THREADS` | Max concurrent active threads (default: `20`) | +| `CLAW_REPLY_HEADER` | Whether to prefix replies with bot name (default: `true`) | +| `CLAW_CLOSE_COMMANDS` | Comma-separated close commands (default: `\close,close,close thread,...`) | +| `CLAW_TRIGGER_REACTION` | Emoji reaction name to trigger Claw (default: `onit`) | +| `CLAW_REACTION_CHANNELS` | Comma-separated channel IDs where reaction triggers are monitored | + +For each required field: +- If it has a placeholder value (like ``), ask the user for the real value and update `app.yaml`. +- If it references a secret (`valueFrom`), do NOT ask for the secret value — that is handled in Phase 4. +- **Validate `ANTHROPIC_BASE_URL` workspace ID matches the authenticated workspace.** Extract the workspace ID from the URL (the numeric prefix in `https://.ai-gateway.cloud.databricks.com/anthropic`) and compare it to the actual workspace ID by running: + ```bash + WORKSPACE_HOST=$(databricks auth describe 2>&1 | awk '/^Host:/{print $2; exit}' | sed 's:/*$::') + TOKEN=$(databricks auth token --host "$WORKSPACE_HOST" 2>/dev/null | python3 -c "import sys,json; print(json.load(sys.stdin)['access_token'])") + ACTUAL_WS_ID=$(curl -sI "${WORKSPACE_HOST}/api/2.0/clusters/spark-versions" -H "Authorization: Bearer ${TOKEN}" | grep -i '^x-databricks-org-id:' | awk '{print $2}' | tr -d '\r') + URL_WS_ID=$(grep 'ai-gateway.cloud.databricks.com' app.yaml | sed 's|.*https://||' | sed 's|\..*||') + echo "Workspace ID from auth: ${ACTUAL_WS_ID}" + echo "Workspace ID from URL: ${URL_WS_ID}" + if [ "$ACTUAL_WS_ID" = "$URL_WS_ID" ]; then echo "✓ MATCH"; else echo "✗ MISMATCH — update ANTHROPIC_BASE_URL in app.yaml"; fi + ``` + If they don't match, update `app.yaml` with the correct workspace ID and STOP until fixed. +- For `CLAW_MEMORY_VOLUME_PATH`, confirm the catalog name matches `var.catalog_name` in `databricks.yml`. + +### 2c. Check databricks.yml variables + +Read `databricks.yml` and confirm the variable defaults make sense for the user's workspace: +- `ai_gateway_endpoint` — must match the `ANTHROPIC_MODEL` value in `app.yaml` +- `catalog_name` — must match the catalog in `CLAW_MEMORY_VOLUME_PATH` +- `slack_uc_connection` — must match `SLACK_UC_CONNECTION` in `app.yaml` +- `secret_scope` — the Databricks secret scope name (default: `claw`) + +Ask the user to confirm or update these. + +--- + +## Phase 3 — Slack UC Connection Verification + +The person running this setup may not have admin access to the Slack app itself. +Instead of verifying Slack scopes manually, we validate that the UC connection is configured and can authenticate with Slack. + +### 3a. Verify the Slack UC Connection exists + +Run: +``` +databricks connections get +``` + +Where `` is the value from `app.yaml` (e.g. `slack`). + +If the connection does not exist, guide the user to create it in the Databricks UI: +1. Go to Catalog > External Connections > Create Connection +2. Connection type: HTTP +3. Host: `https://slack.com` +4. Auth type: Bearer Token +5. Token: their Slack Bot User OAuth Token (from the Slack app settings) + +### 3b. Test the Slack UC Connection + +Test the connection by calling `auth.test` through the Databricks external-function endpoint. +The Slack client in this codebase uses `POST /api/2.0/external-function` with the connection name. + +Run this curl command (replacing placeholders with values from `databricks auth describe` and `app.yaml`): + +```bash +WORKSPACE_HOST=$(databricks auth describe 2>&1 | awk '/^Host:/{print $2; exit}' | sed 's:/*$::') +SLACK_CONN=$(grep -A1 'SLACK_UC_CONNECTION' app.yaml | grep 'value:' | awk '{print $2}' | tr -d '"') +TOKEN=$(databricks auth token --host "$WORKSPACE_HOST" 2>/dev/null | python3 -c "import sys,json; print(json.load(sys.stdin)['access_token'])") + +curl -s -X POST "${WORKSPACE_HOST}/api/2.0/external-function" \ + -H "Authorization: Bearer ${TOKEN}" \ + -H "Content-Type: application/json" \ + -d "{\"connection_name\": \"${SLACK_CONN}\", \"method\": \"POST\", \"path\": \"/auth.test\", \"json\": \"{}\"}" | python3 -m json.tool +``` + +The response should contain `"ok": true` and show the bot's `user_id`, `team`, and `bot_id`. + +If it fails: +- Connection not found → go back to 3b +- `invalid_auth` → the Slack bot token in the UC connection is wrong or expired +- Network error → check the connection host is `https://slack.com` + +STOP and confirm the test passes before proceeding. + +--- + +## Phase 4 — Secrets Verification + +The app needs these secrets. The secret scope is defined in `databricks.yml` variable `secret_scope` (default: `claw`). + +| Secret Key | Purpose | +|---|---| +| `user-pat` | Databricks Personal Access Token — used for AI Gateway auth and UC connection proxy calls | + +### 4a. Ensure the secret scope exists + +``` +databricks secrets list-scopes | grep +``` + +If the scope does not exist: +``` +databricks secrets create-scope +``` + +### 4b. Check each secret exists + +For each secret key, check if it exists (this returns metadata only, NOT the value): +``` +databricks secrets list-secrets +``` + +### 4c. Set missing secrets + +For any missing secret, instruct the user to run: +``` +databricks secrets put-secret +``` + +This opens an interactive editor — the value is never echoed or logged. The user pastes the value there. + +For the `user-pat` secret specifically: this is a Databricks Personal Access Token. The user can create one at: +`https:///settings/user/developer/access-tokens` + +CRITICAL: NEVER ask the user to paste or type their token into this conversation. +Always use `databricks secrets put-secret` which handles input securely. + +--- + +## Phase 5 — Deploy + +### 5a. Full deploy + +Run the end-to-end deployment (install deps, validate bundle, deploy, run migrations, start app): + +``` +make full-deploy +``` + +If it fails, read the error output and help the user fix the issue. Common problems: +- Validation failure → misconfigured variable or missing resource in `databricks.yml` +- Deploy failure → permissions or resource conflicts +- Migration failure → Lakebase not yet provisioned (retry after a minute) +- Start failure → app configuration issue in `app.yaml` + +### 5b. Verify deployment + +Check that the app is running: +``` +databricks apps get claw-app +``` + +The status should show the app is ACTIVE or RUNNING. + +Then tail the logs briefly to confirm startup: +``` +databricks apps get-logs claw-app --limit 50 +``` + +Look for: +- `SlackPoller started` — confirms the poller is running +- `Resolved Slack user_id=...` — confirms Slack auth worked +- No stack traces or startup errors + +If there are errors, read them and help the user fix the issue before proceeding. + +--- + +## Phase 6 — Smoke Test + +### 6a. Identify the DM channel + +From the deployment logs, the bot should have resolved its own user_id and opened a self-DM channel. +Tell the user: + +1. Open Slack +2. Go to your DM with yourself (search for your own name) +3. Send a message: `@claw hello, are you there?` +4. Wait 10-15 seconds (the poller runs every `POLL_INTERVAL_SECONDS`) +5. Claw should reply in a thread + +### 6b. Confirm success + +Ask the user to confirm they received a reply. If they did — setup is complete! + +If not, check: +- App logs for errors: `databricks apps get-logs claw-app --limit 100` +- That the Slack bot is in the workspace +- That the `SLACK_UC_CONNECTION` points to the right Slack bot token +- That the `ANTHROPIC_BASE_URL` and `ANTHROPIC_MODEL` are correct for their AI Gateway + +--- + +## Completion + +When all phases pass, summarize: +- Workspace host +- App name: `claw-app` +- Slack UC connection name +- Memory volume path +- AI Gateway endpoint + +Remind the user: +- To update code and redeploy: `make full-deploy` +- To customize personality: edit `claw/CLAUDE.md` +- To add MCP tools: use the `/add-external-mcp`, `/add-managed-mcp`, or `/add-custom-mcp` skills +- To add skills: create `.claude/skills//SKILL.md` diff --git a/Makefile b/Makefile index 690f36b..f67e4c2 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: help install test lint requirements migrate deploy-dev deploy-prod deploy-destroy clean +.PHONY: help install test lint requirements migrate full-deploy deploy-destroy clean help: ## Show this help @grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf " %-15s %s\n", $$1, $$2}' @@ -39,12 +39,8 @@ run: ## Run the app locally (requires .env) # ── Deploy ──────────────────────────────────────────────────────────────────── -deploy-dev: ## Full deploy to dev (build + validate + deploy + migrate + start) - bash scripts/deploy.sh full --target dev - -deploy-prod: ## Full deploy to prod (requires confirmation) - @read -p "Deploy to PROD? [y/N] " confirm && [ "$$confirm" = "y" ] || exit 1 - bash scripts/deploy.sh full --target prod +full-deploy: ## Full deploy (build + validate + deploy + migrate + start) + bash scripts/deploy.sh full deploy-destroy: ## Tear down the Databricks App databricks bundle destroy