diff --git a/.claude/skills/add-custom-mcp/SKILL.md b/.claude/skills/add-custom-mcp/SKILL.md index ebd493f..4619cac 100644 --- a/.claude/skills/add-custom-mcp/SKILL.md +++ b/.claude/skills/add-custom-mcp/SKILL.md @@ -45,4 +45,4 @@ Print: - Server name: `` - Server URL: `` - Note: The app's service principal token will be sent automatically as `Authorization: Bearer ` on every request. The target server must accept this header (standard MCP auth). -- Reminder: Run `make deploy` to apply the change. +- Reminder: Run `make full-deploy` to apply the change. diff --git a/.claude/skills/add-external-mcp/SKILL.md b/.claude/skills/add-external-mcp/SKILL.md index fc1af2e..b49eb36 100644 --- a/.claude/skills/add-external-mcp/SKILL.md +++ b/.claude/skills/add-external-mcp/SKILL.md @@ -94,7 +94,7 @@ Updated app.yaml: Proxied URL: {DATABRICKS_HOST}/api/2.0/mcp/external/{connection_name} -Next step: run `make deploy` to push the updated config to your Databricks App. +Next step: run `make full-deploy` to push the updated config to your Databricks App. ``` --- diff --git a/.claude/skills/add-managed-mcp/SKILL.md b/.claude/skills/add-managed-mcp/SKILL.md index b98e586..a0b8ee4 100644 --- a/.claude/skills/add-managed-mcp/SKILL.md +++ b/.claude/skills/add-managed-mcp/SKILL.md @@ -18,7 +18,7 @@ This skill helps you configure Databricks-managed MCP servers by: 2. Discovering available resources via the Databricks CLI 3. Letting you select specific resources 4. Updating `app.yaml` with the correct `UC_MANAGED_MCPS` token(s) -5. Reminding you to run `make deploy` to apply changes +5. Reminding you to run `make full-deploy` to apply changes The supported MCP types and their token formats are: @@ -222,7 +222,7 @@ Added managed MCP server(s): Current UC_MANAGED_MCPS value: genie:abc123,genie:01ef1a2b3c4d5678,genie:01ef9a8b7c6d5432 -➡️ Run `make deploy` to apply these changes to your Databricks App. +➡️ Run `make full-deploy` to apply these changes to your Databricks App. ``` --- @@ -240,6 +240,6 @@ Current UC_MANAGED_MCPS value: ## Guardrails - Do NOT modify any file other than `app.yaml`. -- Do NOT run `make deploy` automatically — only remind the user. +- Do NOT run `make full-deploy` automatically — only remind the user. - Always confirm with the user before writing changes. - Always show the full updated `UC_MANAGED_MCPS` value so the user can verify. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..e90b779 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,80 @@ +# Contributing to databricks-claw + +## Prerequisites + +- Python >= 3.11 +- [uv](https://docs.astral.sh/uv/) (Python package manager) +- [Databricks CLI](https://docs.databricks.com/dev-tools/cli/install.html) (authenticated against your workspace) + +## Setup + +```bash +git clone https://github.com/IceRhymers/databricks-claw.git +cd databricks-claw +make install +``` + +This runs `uv sync --all-groups` to install all dependencies including dev tools. + +## Running tests + +```bash +uv run pytest +``` + +Or with coverage: + +```bash +make test-cov +``` + +Linting and type checks: + +```bash +make lint +``` + +## Development workflow + +This project follows an **openspec-TDD workflow**. Before implementing any feature or fix: + +1. Write an openspec (specification) first +2. Write tests that validate the spec +3. Implement until tests pass + +See the `.claude/skills/openspec-tdd/` skill for the full workflow details. + +## Submitting a PR + +1. Create a feature branch from `master`: + ```bash + git checkout -b / + ``` + Branch name types: `feat/`, `fix/`, `chore/`, `refactor/`, `docs/` + +2. Make your changes and ensure tests pass (`uv run pytest`) + +3. Push and open a PR: + ```bash + git push -u origin + gh pr create --title ": short description" --body "..." + ``` + +4. PR description should include: + - What changed and why + - Link to related issue(s) if applicable + - Test plan or confirmation that tests pass + +## Deploying + +After merging, deploy with: + +```bash +make full-deploy +``` + +See `scripts/deploy.sh` for the full deploy pipeline (build, validate, deploy, migrate, start, app-deploy). + +## Working with forks + +If you're contributing from a fork, use the `sync-upstream` skill (`.claude/skills/sync-upstream/`) to keep your fork's master in sync with the upstream repo before branching. diff --git a/README.md b/README.md index c0c9976..0a2ce81 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ You (@claw in a Slack DM thread) │ ▼ Databricks App (FastAPI + Claude Agent SDK) - │ M2M token from your Client ID + Secret + │ Your PAT forwarded via X-Forwarded-Access-Token │ ├── UC Connection → Slack (send responses back) ├── UC Connection → any other external MCP tool @@ -36,16 +36,15 @@ Default: thread-in-DM-with-yourself. Two layers: - *Within-thread*: Full conversation continuity via Claude Agent SDK session IDs keyed on Slack `thread_ts`. Stored in Lakebase. -- *Cross-thread*: `MEMORY.md` on a UC Volume — the agent reads and writes facts it wants to remember across sessions. Same pattern as NanoClaw. +- *Cross-thread*: `MEMORY.md` on a UC Volume — the agent reads and writes facts it wants to remember across sessions. ## Auth -You provide your own Databricks service principal Client ID + Secret as secrets in the DAB config. The app generates M2M tokens from these to: -- Access your UC connections (Slack, other tools) -- Authenticate the Claude Agent SDK subprocess against Databricks AI Gateway -- Call the Databricks SDK (Lakebase, UC Volume, connections) +The app runs as a Databricks App with its own service principal (auto-injected by the platform). You provide a Personal Access Token (PAT) as a Databricks Secret — this PAT is forwarded via `X-Forwarded-Access-Token` on UC connection proxy calls so that external services (Slack, other MCP tools) resolve to *your* identity, not the app's SP. -This means the app runs with *your* identity — the one that has access to your UC connections — not the app's default service principal. +- The app's SP token handles Databricks SDK calls (Lakebase, UC Volume) +- Your PAT is forwarded to UC connection proxies for user-scoped access +- The AI Gateway endpoint is accessed via the app's SP credentials ## Customization @@ -63,15 +62,14 @@ databricks.yml ← workspace, AI Gateway endpoint, UC connection na - Databricks workspace with AI Gateway endpoint configured - Slack app with bot token (`chat:write`, `app_mentions:read`, `im:write`) - Slack UC connection set up in Unity Catalog -- Databricks service principal Client ID + Secret +- Databricks Personal Access Token (PAT) for the deploying user ### Configure 1. Copy `app.example.yml` to `app.yaml` and fill in your values -2. Add secrets to your workspace: +2. Add your PAT as a secret in your workspace: ```bash - databricks secrets put-secret claw claw-client-id --string-value - databricks secrets put-secret claw claw-client-secret --string-value + databricks secrets put-secret claw claw-user-pat --string-value ``` ### Deploy diff --git a/design.md b/design.md index 1cf57d0..025cf27 100644 --- a/design.md +++ b/design.md @@ -32,38 +32,32 @@ POST response back to Slack via UC connection HTTP proxy --- -## Auth Model: M2M Client Credentials +## Auth Model: SP + PAT Forwarding -OBO tokens only exist within the scope of an incoming HTTP request — they're injected by Databricks into request headers, not available at startup or in background tasks. The app's default service principal won't have access to *your* UC connections (they're scoped to the deploying user). - -**Solution**: User stores their own Client ID + Secret as Databricks Secrets in the DAB config. The app uses these for M2M token generation via `WorkspaceClient(client_id=..., client_secret=...)`. +The app runs as a Databricks App with an auto-injected service principal (SP). The SP handles all Databricks SDK calls (Lakebase, UC Volume, connections list). However, UC connection proxies need to resolve to the *deploying user's* identity — so the user provides a Personal Access Token (PAT) stored as a Databricks Secret. ```python -ws = WorkspaceClient( - host=config.databricks_host, - client_id=config.client_id, # from DATABRICKS_CLIENT_ID env var - client_secret=config.client_secret, # from DATABRICKS_CLIENT_SECRET env var -) -token = ws.config.token # M2M token — used as OBO proxy header +# SP client — auto-configured by Databricks Apps runtime +ws = WorkspaceClient(host=config.DATABRICKS_HOST) +sp_token = ws.config.token + +# User PAT — stored as a Databricks Secret, read from env +user_pat = config.DATABRICKS_USER_PAT ``` -This token is: -- Used for `ws.connections.get()` / `ws.database.*` SDK calls -- Forwarded to UC connection MCP proxy: `Authorization: Bearer {token}` + `X-Forwarded-Access-Token: {token}` -- Passed as `ANTHROPIC_AUTH_TOKEN` / `ANTHROPIC_API_KEY` in `sdk_env` +How tokens are used: +- SP token: `Authorization: Bearer {sp_token}` — authenticates SDK calls and the proxy request itself +- User PAT: `X-Forwarded-Access-Token: {user_pat}` — tells the UC proxy to use the user's identity for upstream calls (Slack, external MCP tools) +- AI Gateway: SP credentials (auto-injected) + `ANTHROPIC_AUTH_TOKEN` / `ANTHROPIC_API_KEY` in `sdk_env` DAB secrets config: ```yaml config: env: - - name: DATABRICKS_CLIENT_ID - value_from: - secret: - name: claw-client-id - - name: DATABRICKS_CLIENT_SECRET + - name: DATABRICKS_USER_PAT value_from: secret: - name: claw-client-secret + name: claw-user-pat ``` --- @@ -111,7 +105,7 @@ Default interaction pattern: This requires minimal Slack infrastructure: one bot, one DM. No channel setup needed. -Additional patterns (like NanoClaw groups) can be layered on via config. +Additional interaction patterns (e.g. multi-user channels) can be layered on via config. --- @@ -134,11 +128,11 @@ On each message: look up `thread_ts` → if session exists, resume it; else star ### Layer 2 — Cross-thread memory (MEMORY.md on UC Volume) -Agent reads and writes `MEMORY.md` on a UC Volume — same pattern as NanoClaw. Loaded into every session via the system prompt. Agent explicitly writes facts it wants to persist ("let me save this to memory"). +Agent reads and writes `MEMORY.md` on a UC Volume. Loaded into every session via the system prompt. Agent explicitly writes facts it wants to persist ("let me save this to memory"). Volume path: `/Volumes/{catalog}/{schema}/claw-memory/MEMORY.md` -The agent also manages typed memory files in `/Volumes/{catalog}/{schema}/claw-memory/` (user facts, project notes, etc.) — identical to NanoClaw's `/home/node/.claude/projects/*/memory/` structure. +The agent also manages typed memory files in `/Volumes/{catalog}/{schema}/claw-memory/` (user facts, project notes, etc.). --- @@ -231,31 +225,9 @@ databricks-claw/ --- -## databricks.yml (planned) +## databricks.yml -```yaml -bundle: - name: databricks-claw - -variables: - ai_gateway_endpoint: - description: AI Gateway endpoint name - default: databricks-claude-sonnet-4-5 - catalog_name: - description: Unity Catalog catalog for volumes - slack_uc_connection: - description: UC connection name for Slack - -include: - - resources/*.yml - -targets: - dev: - mode: development - default: true - prod: - mode: production -``` +The DAB bundle config lives at `databricks.yml` in the repo root. It defines variables (`ai_gateway_endpoint`, `catalog_name`, `slack_uc_connection`, `secret_scope`, `name_prefix`), includes `resources/*.yml`, and declares `dev` (default) and `prod` targets. --- diff --git a/tests/test_skills.py b/tests/test_skills.py index 331355c..401d4b8 100644 --- a/tests/test_skills.py +++ b/tests/test_skills.py @@ -27,9 +27,9 @@ def test_mentions_https(self): content = self.SKILL_PATH.read_text() assert "https://" in content - def test_mentions_make_deploy(self): + def test_mentions_make_full_deploy(self): content = self.SKILL_PATH.read_text() - assert "make deploy" in content + assert "make full-deploy" in content def test_mentions_authorization(self): content = self.SKILL_PATH.read_text() @@ -71,8 +71,8 @@ def test_mentions_uc_functions(self, skill_content: str): def test_mentions_uc_managed_mcps(self, skill_content: str): assert "UC_MANAGED_MCPS" in skill_content - def test_mentions_make_deploy(self, skill_content: str): - assert "make deploy" in skill_content + def test_mentions_make_full_deploy(self, skill_content: str): + assert "make full-deploy" in skill_content class TestAddExternalMcpSkill: @@ -96,5 +96,5 @@ def test_references_uc_mcp_connections(self, skill_content: str): def test_references_databricks_connections_list(self, skill_content: str): assert "databricks connections list" in skill_content - def test_references_make_deploy(self, skill_content: str): - assert "make deploy" in skill_content + def test_references_make_full_deploy(self, skill_content: str): + assert "make full-deploy" in skill_content