|
9 | 9 |
|
10 | 10 | --- |
11 | 11 |
|
12 | | -## Installation |
| 12 | +### What is the Patterns Devkit? |
| 13 | + |
| 14 | +The Patterns Devkit is a CLI and lightweight SDK to build, version, and deploy data graphs made of reusable SQL and Python nodes. It helps you: |
| 15 | +- Scaffold apps (graphs) and nodes quickly |
| 16 | +- Define connections between nodes and storage tables in `graph.yml` |
| 17 | +- Manage secrets and configuration |
| 18 | +- Upload, list, and trigger runs in the Patterns platform |
| 19 | + |
| 20 | +Documentation: `https://www.patterns.app/docs/devkit` |
| 21 | + |
| 22 | +### Features |
| 23 | +- Create graphs and nodes (`python`, `sql`, subgraphs) from the CLI |
| 24 | +- Describe graph topology declaratively in `graph.yml` |
| 25 | +- Write nodes using `patterns.Table`, `patterns.Parameter`, and `patterns.State` |
| 26 | +- Manage secrets, auth, and uploads to the Patterns platform |
| 27 | +- Trigger and inspect graphs remotely |
| 28 | + |
| 29 | +### Installation |
13 | 30 |
|
14 | 31 | `pip install patterns-devkit` |
15 | 32 |
|
16 | | -## Usage |
| 33 | +### Quickstart: Build a Leads Ingestion and Scoring Graph |
| 34 | + |
| 35 | +1) Create an app (graph) |
17 | 36 |
|
18 | | -`patterns create graph mygraph` |
| 37 | +```bash |
| 38 | +patterns create app my-leads-app |
| 39 | +cd my-leads-app |
| 40 | +``` |
| 41 | + |
| 42 | +This creates: |
| 43 | + |
| 44 | +```text |
| 45 | +my-leads-app/ |
| 46 | + graph.yml |
| 47 | +``` |
19 | 48 |
|
20 | | -This will create an empty patterns graph: |
| 49 | +2) Add two Python nodes |
21 | 50 |
|
| 51 | +```bash |
| 52 | +patterns create node --title "Ingest Leads" ingest_leads.py |
| 53 | +patterns create node --title "Score Leads" score_leads.py |
22 | 54 | ``` |
23 | | -mygraph/ |
| 55 | + |
| 56 | +This adds: |
| 57 | + |
| 58 | +```text |
| 59 | +my-leads-app/ |
24 | 60 | graph.yml |
| 61 | + ingest_leads.py |
| 62 | + score_leads.py |
| 63 | +``` |
| 64 | + |
| 65 | +3) Wire the graph in `graph.yml` |
| 66 | + |
| 67 | +Open `graph.yml` and connect node inputs/outputs to tables: |
| 68 | + |
| 69 | +```yaml |
| 70 | +title: Leads Scoring |
| 71 | + |
| 72 | +stores: |
| 73 | + - table: raw_leads |
| 74 | + - table: scored_leads |
| 75 | + |
| 76 | +functions: |
| 77 | + - node_file: ingest_leads.py |
| 78 | + title: Ingest Leads |
| 79 | + trigger: manual |
| 80 | + outputs: |
| 81 | + leads: raw_leads |
| 82 | + |
| 83 | + - node_file: score_leads.py |
| 84 | + title: Score Leads |
| 85 | + inputs: |
| 86 | + leads: raw_leads |
| 87 | + outputs: |
| 88 | + scored: scored_leads |
25 | 89 | ``` |
26 | 90 |
|
27 | | -Create a new python node: |
| 91 | +4) Implement the nodes |
| 92 | +
|
| 93 | +`ingest_leads.py` (writes raw leads): |
| 94 | + |
| 95 | +```python |
| 96 | +from patterns import Table, Parameter |
| 97 | +
|
| 98 | +def run(): |
| 99 | + # Optionally parameterize where to ingest from |
| 100 | + source = Parameter("leads_source", description="Lead source label", type=str, default="marketing_form") |
| 101 | +
|
| 102 | + raw_leads = Table("raw_leads", mode="w", description="Raw inbound leads") |
| 103 | + # Provide schema and helpful ordering for downstream streaming if desired |
| 104 | + raw_leads.init( |
| 105 | + schema={"id": "Text", "email": "Text", "source": "Text", "created_at": "Datetime"}, |
| 106 | + unique_on="id", |
| 107 | + add_created="created_at", |
| 108 | + ) |
28 | 109 |
|
| 110 | + # Replace this with real ingestion (API/CSV/etc.) |
| 111 | + sample = [ |
| 112 | + {"id": "L-001", "email": "user1@example.com", "source": source}, |
| 113 | + {"id": "L-002", "email": "user2@corp.com", "source": source}, |
| 114 | + {"id": "L-003", "email": "ceo@enterprise.com","source": source}, |
| 115 | + ] |
| 116 | + raw_leads.upsert(sample) |
29 | 117 | ``` |
30 | | -cd mygraph |
31 | | -patterns create node mynode.py |
| 118 | + |
| 119 | +`score_leads.py` (reads raw leads, writes scored leads): |
| 120 | + |
| 121 | +```python |
| 122 | +from patterns import Table |
| 123 | +
|
| 124 | +def lead_score(email: str) -> float: |
| 125 | + # Simple heuristic: enterprise domains score higher |
| 126 | + domain = email.split("@")[-1].lower() |
| 127 | + if domain.endswith("enterprise.com"): |
| 128 | + return 0.95 |
| 129 | + if domain.endswith("corp.com"): |
| 130 | + return 0.8 |
| 131 | + return 0.4 |
| 132 | +
|
| 133 | +def run(): |
| 134 | + raw = Table("raw_leads") # read mode by default |
| 135 | + scored = Table("scored_leads", "w") # write mode |
| 136 | + scored.init( |
| 137 | + schema={"id": "Text", "email": "Text", "score": "Float", "created_at": "Datetime"}, |
| 138 | + unique_on="id", |
| 139 | + add_created="created_at", |
| 140 | + ) |
| 141 | +
|
| 142 | + rows = raw.read() # list[dict] or dataframe if configured |
| 143 | + for r in rows: |
| 144 | + r["score"] = lead_score(r["email"]) |
| 145 | + scored.upsert(rows) |
32 | 146 | ``` |
33 | 147 |
|
| 148 | +5) Visualize the example graph topology |
| 149 | + |
| 150 | +```mermaid |
| 151 | +flowchart TD |
| 152 | + A["Ingest Leads (Python)"] -->|raw_leads| B["Score Leads (Python)"] |
| 153 | + B -->|scored_leads| C[(scored_leads)] |
34 | 154 | ``` |
35 | | -mygraph/ |
36 | | - graph.yml |
37 | | - mynode.py |
| 155 | + |
| 156 | +6) Authenticate and upload |
| 157 | + |
| 158 | +- Sign up or sign in at `https://studio.patterns.app` |
| 159 | +- Authenticate the CLI: |
| 160 | + |
| 161 | +```bash |
| 162 | +patterns login |
38 | 163 | ``` |
39 | 164 |
|
40 | | -## Upload |
| 165 | +- Upload your graph: |
41 | 166 |
|
42 | | -To deploy a graph, you must sign up for a [patterns.app](https://studio.patterns.app) |
43 | | -account and login to authenticate the cli: |
| 167 | +```bash |
| 168 | +patterns upload |
| 169 | +``` |
44 | 170 |
|
45 | | -`patterns login` |
| 171 | +7) Trigger runs |
46 | 172 |
|
47 | | -Then you can upload your graph: |
| 173 | +```bash |
| 174 | +# Trigger any node by title or id (see list commands below to find ids) |
| 175 | +patterns trigger node "Ingest Leads" |
| 176 | +patterns trigger node "Score Leads" |
| 177 | +``` |
48 | 178 |
|
49 | | -`patterns upload` |
| 179 | +### Command overview |
50 | 180 |
|
51 | | -## Other commands |
| 181 | +- `patterns create app <dir>`: scaffold a new app directory with `graph.yml` |
| 182 | +- `patterns create node <file.py|file.sql|graph.yml>`: add a function node (Python/SQL/subgraph) |
| 183 | +- `patterns create node --type table <table_name>`: add a table store |
| 184 | +- `patterns create secret <name> <value>`: create an organization secret |
| 185 | +- `patterns upload`: upload current app to the platform |
| 186 | +- `patterns list apps|nodes|webhooks|versions`: list resources |
| 187 | +- `patterns trigger node <title|id>`: manually trigger a node |
| 188 | +- `patterns download <app>`: download app contents from the platform |
| 189 | +- `patterns update`: update local metadata from remote |
| 190 | +- `patterns delete <resource>`: delete remote resources |
| 191 | +- `patterns config --json`: print CLI configuration |
| 192 | +- `patterns login` / `patterns logout`: authenticate the CLI |
52 | 193 |
|
53 | | -You can see the full list of available cli commands: |
| 194 | +See full help: |
54 | 195 |
|
55 | 196 | ``` |
56 | 197 | patterns --help |
57 | 198 | ``` |
| 199 | + |
| 200 | +### Node development APIs (Python) |
| 201 | +Nodes use a small SDK provided by the platform when running: |
| 202 | +- `Table(name, mode="r"|"w")`: read/write table abstraction. Common methods: |
| 203 | + - `init(schema=..., unique_on=..., add_created=..., add_monotonic_id=...)` |
| 204 | + - `read(as_format="records"|"dataframe", chunksize=...)` |
| 205 | + - `read_sql(sql, ...)` |
| 206 | + - `append(records)`, `upsert(records)`, `replace(records)`, `truncate()`, `flush()` |
| 207 | +- `Parameter(name, description=None, type=str|int|float|bool|datetime|date|list, default=MISSING)`: declare runtime parameters |
| 208 | +- `State`: simple key-value state for long-running or iterative jobs |
| 209 | + |
| 210 | +For more, visit the docs: `https://docs.patterns.app/docs/node-development/python/` |
| 211 | + |
| 212 | +### Tips |
| 213 | +- Prefer explicit schemas on write tables via `Table.init` to control types and indexes |
| 214 | +- Use `unique_on` and `upsert` to deduplicate reliably |
| 215 | +- Add `add_created` or `add_monotonic_id` to enable robust downstream streaming |
| 216 | +- Keep node code small, composable, and parameterized for reuse |
| 217 | + |
| 218 | +### License |
| 219 | + |
| 220 | +BSD-3-Clause (see `LICENSE`) |
0 commit comments