Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .claude/skills/add-custom-mcp/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,4 +45,4 @@ Print:
- Server name: `<name>`
- Server URL: `<url>`
- Note: The app's service principal token will be sent automatically as `Authorization: Bearer <token>` on every request. The target server must accept this header (standard MCP auth).
- Reminder: Run `make deploy` to apply the change.
- Reminder: Run `make full-deploy` to apply the change.
2 changes: 1 addition & 1 deletion .claude/skills/add-external-mcp/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ Updated app.yaml:

Proxied URL: {DATABRICKS_HOST}/api/2.0/mcp/external/{connection_name}

Next step: run `make deploy` to push the updated config to your Databricks App.
Next step: run `make full-deploy` to push the updated config to your Databricks App.
```

---
Expand Down
6 changes: 3 additions & 3 deletions .claude/skills/add-managed-mcp/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ This skill helps you configure Databricks-managed MCP servers by:
2. Discovering available resources via the Databricks CLI
3. Letting you select specific resources
4. Updating `app.yaml` with the correct `UC_MANAGED_MCPS` token(s)
5. Reminding you to run `make deploy` to apply changes
5. Reminding you to run `make full-deploy` to apply changes

The supported MCP types and their token formats are:

Expand Down Expand Up @@ -222,7 +222,7 @@ Added managed MCP server(s):
Current UC_MANAGED_MCPS value:
genie:abc123,genie:01ef1a2b3c4d5678,genie:01ef9a8b7c6d5432

➡️ Run `make deploy` to apply these changes to your Databricks App.
➡️ Run `make full-deploy` to apply these changes to your Databricks App.
```

---
Expand All @@ -240,6 +240,6 @@ Current UC_MANAGED_MCPS value:
## Guardrails

- Do NOT modify any file other than `app.yaml`.
- Do NOT run `make deploy` automatically — only remind the user.
- Do NOT run `make full-deploy` automatically — only remind the user.
- Always confirm with the user before writing changes.
- Always show the full updated `UC_MANAGED_MCPS` value so the user can verify.
80 changes: 80 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# Contributing to databricks-claw

## Prerequisites

- Python >= 3.11
- [uv](https://docs.astral.sh/uv/) (Python package manager)
- [Databricks CLI](https://docs.databricks.com/dev-tools/cli/install.html) (authenticated against your workspace)

## Setup

```bash
git clone https://github.com/IceRhymers/databricks-claw.git
cd databricks-claw
make install
```

This runs `uv sync --all-groups` to install all dependencies including dev tools.

## Running tests

```bash
uv run pytest
```

Or with coverage:

```bash
make test-cov
```

Linting and type checks:

```bash
make lint
```

## Development workflow

This project follows an **openspec-TDD workflow**. Before implementing any feature or fix:

1. Write an openspec (specification) first
2. Write tests that validate the spec
3. Implement until tests pass

See the `.claude/skills/openspec-tdd/` skill for the full workflow details.

## Submitting a PR

1. Create a feature branch from `master`:
```bash
git checkout -b <type>/<short-description>
```
Branch name types: `feat/`, `fix/`, `chore/`, `refactor/`, `docs/`

2. Make your changes and ensure tests pass (`uv run pytest`)

3. Push and open a PR:
```bash
git push -u origin <branch-name>
gh pr create --title "<type>: short description" --body "..."
```

4. PR description should include:
- What changed and why
- Link to related issue(s) if applicable
- Test plan or confirmation that tests pass

## Deploying

After merging, deploy with:

```bash
make full-deploy
```

See `scripts/deploy.sh` for the full deploy pipeline (build, validate, deploy, migrate, start, app-deploy).

## Working with forks

If you're contributing from a fork, use the `sync-upstream` skill (`.claude/skills/sync-upstream/`) to keep your fork's master in sync with the upstream repo before branching.
20 changes: 9 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ You (@claw in a Slack DM thread)
Databricks App (FastAPI + Claude Agent SDK)
M2M token from your Client ID + Secret
Your PAT forwarded via X-Forwarded-Access-Token
├── UC Connection → Slack (send responses back)
├── UC Connection → any other external MCP tool
Expand All @@ -36,16 +36,15 @@ Default: thread-in-DM-with-yourself.
Two layers:

- *Within-thread*: Full conversation continuity via Claude Agent SDK session IDs keyed on Slack `thread_ts`. Stored in Lakebase.
- *Cross-thread*: `MEMORY.md` on a UC Volume — the agent reads and writes facts it wants to remember across sessions. Same pattern as NanoClaw.
- *Cross-thread*: `MEMORY.md` on a UC Volume — the agent reads and writes facts it wants to remember across sessions.

## Auth

You provide your own Databricks service principal Client ID + Secret as secrets in the DAB config. The app generates M2M tokens from these to:
- Access your UC connections (Slack, other tools)
- Authenticate the Claude Agent SDK subprocess against Databricks AI Gateway
- Call the Databricks SDK (Lakebase, UC Volume, connections)
The app runs as a Databricks App with its own service principal (auto-injected by the platform). You provide a Personal Access Token (PAT) as a Databricks Secret — this PAT is forwarded via `X-Forwarded-Access-Token` on UC connection proxy calls so that external services (Slack, other MCP tools) resolve to *your* identity, not the app's SP.

This means the app runs with *your* identity — the one that has access to your UC connections — not the app's default service principal.
- The app's SP token handles Databricks SDK calls (Lakebase, UC Volume)
- Your PAT is forwarded to UC connection proxies for user-scoped access
- The AI Gateway endpoint is accessed via the app's SP credentials

## Customization

Expand All @@ -63,15 +62,14 @@ databricks.yml ← workspace, AI Gateway endpoint, UC connection na
- Databricks workspace with AI Gateway endpoint configured
- Slack app with bot token (`chat:write`, `app_mentions:read`, `im:write`)
- Slack UC connection set up in Unity Catalog
- Databricks service principal Client ID + Secret
- Databricks Personal Access Token (PAT) for the deploying user

### Configure

1. Copy `app.example.yml` to `app.yaml` and fill in your values
2. Add secrets to your workspace:
2. Add your PAT as a secret in your workspace:
```bash
databricks secrets put-secret claw claw-client-id --string-value <your-client-id>
databricks secrets put-secret claw claw-client-secret --string-value <your-client-secret>
databricks secrets put-secret claw claw-user-pat --string-value <your-pat>
```

### Deploy
Expand Down
66 changes: 19 additions & 47 deletions design.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,38 +32,32 @@ POST response back to Slack via UC connection HTTP proxy

---

## Auth Model: M2M Client Credentials
## Auth Model: SP + PAT Forwarding

OBO tokens only exist within the scope of an incoming HTTP request — they're injected by Databricks into request headers, not available at startup or in background tasks. The app's default service principal won't have access to *your* UC connections (they're scoped to the deploying user).

**Solution**: User stores their own Client ID + Secret as Databricks Secrets in the DAB config. The app uses these for M2M token generation via `WorkspaceClient(client_id=..., client_secret=...)`.
The app runs as a Databricks App with an auto-injected service principal (SP). The SP handles all Databricks SDK calls (Lakebase, UC Volume, connections list). However, UC connection proxies need to resolve to the *deploying user's* identity — so the user provides a Personal Access Token (PAT) stored as a Databricks Secret.

```python
ws = WorkspaceClient(
host=config.databricks_host,
client_id=config.client_id, # from DATABRICKS_CLIENT_ID env var
client_secret=config.client_secret, # from DATABRICKS_CLIENT_SECRET env var
)
token = ws.config.token # M2M token — used as OBO proxy header
# SP client — auto-configured by Databricks Apps runtime
ws = WorkspaceClient(host=config.DATABRICKS_HOST)
sp_token = ws.config.token

# User PAT — stored as a Databricks Secret, read from env
user_pat = config.DATABRICKS_USER_PAT
```

This token is:
- Used for `ws.connections.get()` / `ws.database.*` SDK calls
- Forwarded to UC connection MCP proxy: `Authorization: Bearer {token}` + `X-Forwarded-Access-Token: {token}`
- Passed as `ANTHROPIC_AUTH_TOKEN` / `ANTHROPIC_API_KEY` in `sdk_env`
How tokens are used:
- SP token: `Authorization: Bearer {sp_token}` — authenticates SDK calls and the proxy request itself
- User PAT: `X-Forwarded-Access-Token: {user_pat}` — tells the UC proxy to use the user's identity for upstream calls (Slack, external MCP tools)
- AI Gateway: SP credentials (auto-injected) + `ANTHROPIC_AUTH_TOKEN` / `ANTHROPIC_API_KEY` in `sdk_env`

DAB secrets config:
```yaml
config:
env:
- name: DATABRICKS_CLIENT_ID
value_from:
secret:
name: claw-client-id
- name: DATABRICKS_CLIENT_SECRET
- name: DATABRICKS_USER_PAT
value_from:
secret:
name: claw-client-secret
name: claw-user-pat
```

---
Expand Down Expand Up @@ -111,7 +105,7 @@ Default interaction pattern:

This requires minimal Slack infrastructure: one bot, one DM. No channel setup needed.

Additional patterns (like NanoClaw groups) can be layered on via config.
Additional interaction patterns (e.g. multi-user channels) can be layered on via config.

---

Expand All @@ -134,11 +128,11 @@ On each message: look up `thread_ts` → if session exists, resume it; else star

### Layer 2 — Cross-thread memory (MEMORY.md on UC Volume)

Agent reads and writes `MEMORY.md` on a UC Volume — same pattern as NanoClaw. Loaded into every session via the system prompt. Agent explicitly writes facts it wants to persist ("let me save this to memory").
Agent reads and writes `MEMORY.md` on a UC Volume. Loaded into every session via the system prompt. Agent explicitly writes facts it wants to persist ("let me save this to memory").

Volume path: `/Volumes/{catalog}/{schema}/claw-memory/MEMORY.md`

The agent also manages typed memory files in `/Volumes/{catalog}/{schema}/claw-memory/` (user facts, project notes, etc.) — identical to NanoClaw's `/home/node/.claude/projects/*/memory/` structure.
The agent also manages typed memory files in `/Volumes/{catalog}/{schema}/claw-memory/` (user facts, project notes, etc.).

---

Expand Down Expand Up @@ -231,31 +225,9 @@ databricks-claw/

---

## databricks.yml (planned)
## databricks.yml

```yaml
bundle:
name: databricks-claw

variables:
ai_gateway_endpoint:
description: AI Gateway endpoint name
default: databricks-claude-sonnet-4-5
catalog_name:
description: Unity Catalog catalog for volumes
slack_uc_connection:
description: UC connection name for Slack

include:
- resources/*.yml

targets:
dev:
mode: development
default: true
prod:
mode: production
```
The DAB bundle config lives at `databricks.yml` in the repo root. It defines variables (`ai_gateway_endpoint`, `catalog_name`, `slack_uc_connection`, `secret_scope`, `name_prefix`), includes `resources/*.yml`, and declares `dev` (default) and `prod` targets.

---

Expand Down
12 changes: 6 additions & 6 deletions tests/test_skills.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,9 @@ def test_mentions_https(self):
content = self.SKILL_PATH.read_text()
assert "https://" in content

def test_mentions_make_deploy(self):
def test_mentions_make_full_deploy(self):
content = self.SKILL_PATH.read_text()
assert "make deploy" in content
assert "make full-deploy" in content

def test_mentions_authorization(self):
content = self.SKILL_PATH.read_text()
Expand Down Expand Up @@ -71,8 +71,8 @@ def test_mentions_uc_functions(self, skill_content: str):
def test_mentions_uc_managed_mcps(self, skill_content: str):
assert "UC_MANAGED_MCPS" in skill_content

def test_mentions_make_deploy(self, skill_content: str):
assert "make deploy" in skill_content
def test_mentions_make_full_deploy(self, skill_content: str):
assert "make full-deploy" in skill_content


class TestAddExternalMcpSkill:
Expand All @@ -96,5 +96,5 @@ def test_references_uc_mcp_connections(self, skill_content: str):
def test_references_databricks_connections_list(self, skill_content: str):
assert "databricks connections list" in skill_content

def test_references_make_deploy(self, skill_content: str):
assert "make deploy" in skill_content
def test_references_make_full_deploy(self, skill_content: str):
assert "make full-deploy" in skill_content
Loading