Discussion: pluggable Backend protocol for chDB integration (alternative to #753)

Cross-referencing #753.

## TL;DR

To let clickhouse-connect users target either a remote ClickHouse server or in-process chDB **with a one-line change** — without forking the client, without losing chDB's zero-copy advantage, and without making clickhouse-connect carry chDB-specific code — this proposal introduces a **pluggable `Backend`** behind a single public `Client` API:

- clickhouse-connect provides a small `Backend` protocol + an entry-point registry.
- `ChdbBackend` lives in the **chdb-io/chdb repo** and self-registers via a Python entry-point.
- **clickhouse-connect ships zero chDB code, zero chDB tests, zero chDB CI dependency.**
- A `supports_zero_copy_arrow` capability flag routes `query_df()` through chDB's in-process Arrow path for end-to-end zero-copy.
- chDB-unique APIs (Python UDFs, `Python()` table function, in-memory vs persistent path management) surface via a `.chdb` extension namespace, not in the base `Client`.
- Tests run on every installed backend via a parametrized fixture; **chDB owns the skip manifest** in its own repo via a second entry-point.
- CI is split: clickhouse-connect CI never touches chDB; chdb-io/chdb CI runs the full clickhouse-connect test suite against `ChdbBackend`.

## Core principle (one sentence)

> **clickhouse-connect provides "one `Client` + a `Backend` protocol + a registry"; the chdb-io/chdb package provides a `ChdbBackend` that implements the protocol and self-registers via Python entry-points. The two repositories are coupled only through the protocol signature — clickhouse-connect ships zero chDB code, zero chDB tests, and zero chDB CI dependency.**

---

The detailed design is folded below — expand the sections you want to dig into.

<details>
<summary>📦 1. Repository boundary — what lives where, and what is the only coupling surface</summary>

```
┌──────────────────────────────────────────┐ ┌──────────────────────────────────────────┐
│ clickhouse-connect repo │ │ chdb-io/chdb repo │
│ github.com/ClickHouse/clickhouse-connect│ │ github.com/chdb-io/chdb │
│ (owned by CH team) │ │ (owned by chDB team) │
├──────────────────────────────────────────┤ ├──────────────────────────────────────────┤
│ driver/ │ │ chdb/ │
│ client.py public semantics │ │ cc_backend.py ChdbBackend │
│ backend.py Backend protocol [new]│ │ cc_extension.py .chdb extension │
│ registry.py entry-point loader[new]│ │ cc_test_profile.py skip manifest │
│ httpbackend.py = old HttpClient, │ │ pyproject.toml │
│ transport split out │◄───┤ [entry-points │
│ __init__.py get_client routing │ │ ."clickhouse_connect.backends"] │
│ │ │ chdb = "chdb.cc_backend:ChdbBackend"│
│ tests/ │ │ [entry-points │
│ integration/ parametrized fixture │ │ ."clickhouse_connect.test_profile"] │
│ contract/ backend-contract suite │ │ chdb = "chdb.cc_test_profile" │
│ │ │ tests/ │
│ ✗ zero chdb code │ │ runs full clickhouse-connect suite │
│ ✗ zero chdb tests │ │ against chdb backend │
│ ✗ CI does not depend on chdb │ │ ✓ CI installs chdb + clickhouse-connect │
└──────────────────────────────────────────┘ └──────────────────────────────────────────┘
 ▲ │
 │ protocol signature + entry-point names │
 └──────────────────────────────────────────────┘
 (the only coupling surface)
```

The coupling surface is intentionally tiny:

- The `Backend` protocol signature (defined in clickhouse-connect).
- Two entry-point group names: `clickhouse_connect.backends`, `clickhouse_connect.test_profile`.

Everything else — chDB version, internal output formats, Python UDF API, file paths, schema quirks — is invisible to clickhouse-connect. The `ChdbBackend` adapter consumes chdb's already-public Python API (`chdb.connect`, `Connection.query`, `Connection.send_query`, `StreamingResult.record_batch`, `to_arrowTable`) and does **not** require any changes to chdb-core.

</details>

<details>
<summary>🔧 2. Refactor and abstraction — before/after, plus the protocol-evolution discipline</summary>

### Before

```
HttpClient(Client)
 ├─ Public API: query / query_df / query_arrow / insert / streaming / ...
 ├─ Semantics: QueryContext, type system, Native parser, settings, error model
 └─ Transport: urllib3 pool, headers, HTTP encoding
```

`HttpClient` is the only Client subclass; semantics and transport are coupled.

### After

```
Client ← single semantics layer, shared by all backends
 ├─ query / query_df / query_arrow / insert / streaming / command ...
 ├─ QueryContext / InsertContext / type system / Native parser
 ├─ settings normalization, unified errors, async wrapper entry
 └─ self.backend: Backend ◄─┐
 │ injected by get_client()
 ┌────────────────────┴──────────────────┐
 │ │
 HttpBackend ChdbBackend
 (clickhouse-connect repo) (chdb-io/chdb repo)
 - raw_query / raw_insert / command - raw_query via chdb.connect()
 - urllib3 pool - query_arrow fast path: zero-copy via
 - supports_zero_copy_arrow = False res.get_memview() (chdb.to_arrowTable)
 (HTTP wire bytes are unavoidable; - query_arrow_stream: native
 Native parser path keeps better pa.RecordBatchReader via
 type fidelity for LowCardinality StreamingResult.record_batch()
 / Enum / nested / Decimal) - insert_arrow via Python() table function
 (zero-copy DataFrame ingest)
 - supports_zero_copy_arrow = True
```

Transport is extracted from `Client` into a `Backend`. Public semantics, type fidelity, settings handling, the error model, and async wrapping all stay in one shared layer.

### Protocol-evolution discipline

To prevent the protocol from churning and breaking registered backends:

1. **Methods are additive only** — never remove, never change a method signature.
2. **Every new method ships with a default implementation** that falls back to `raw_query`, so existing backends keep working without any code change.
3. **Capabilities are declared via `supports_*` flags** and are additive — flag meanings are never repurposed; new capabilities get new flag names.

With this discipline, protocol evolution is a one-way ratchet: clickhouse-connect can add anything, but cannot reach back and break an already-shipped backend.

</details>

<details>
<summary>🧬 3. API design — Backend protocol, capability negotiation, .chdb extension namespace</summary>

### 3.1 Backend protocol (defined in clickhouse-connect)

```python
# clickhouse_connect/driver/backend.py
from typing import Protocol, Iterator, Any

class Backend(Protocol):
 # ── Identity ────────────────────────────────────────────────
 backend_name: str

 # ── Capability flags ────────────────────────────────────────
 # Routing flag with real semantic impact today:
 # True → Client.query_df should prefer the Arrow → pandas path
 # (the backend's query_arrow is genuinely zero-copy in-process)
 # False → Client.query_df should keep using the Native parser path
 # (wire bytes are unavoidable, so Arrow buys nothing; the
 # Native parser preserves richer type fidelity for
 # LowCardinality / Enum / nested / Decimal)
 supports_zero_copy_arrow: bool = False

 # ── Core transport (mandatory, ~thin glue per backend) ──────
 def raw_query(self, sql: str, params, settings, fmt: str) -> bytes: ...
 def raw_stream(self, sql: str, params, settings, fmt: str) -> Iterator[bytes]: ...
 def raw_insert(self, table: str, cols, data, settings, fmt: str) -> None: ...
 def command(self, sql: str, params, settings) -> Any: ...
 def ping(self) -> bool: ...
 def close(self) -> None: ...

 # ── Error mapping (backend-specific exceptions never leak) ──
 def map_error(self, exc: BaseException) -> Exception: ...

 # ── Server metadata ─────────────────────────────────────────
 @property
 def server_version(self) -> str: ...
 @property
 def server_timezone(self) -> str: ...

 # ── Arrow surface (both backends implement; flag controls
 # only whether Client routes query_df through here) ──────
 def query_arrow(self, sql, params, settings) -> "pyarrow.Table": ...
 def query_arrow_stream(self, sql, params, settings) -> "pa.RecordBatchReader": ...
 def insert_arrow(self, table: str, pa_table, settings) -> None: ...
```

| Backend | `supports_zero_copy_arrow` | Why |
|---|---|---|
| `HttpBackend` | `False` | HTTP always crosses wire bytes; the Native parser preserves better type fidelity than an Arrow round-trip would. `query_arrow()` still works (existing HTTP behavior unchanged) — but `query_df()` keeps using the Native parser path. |
| `ChdbBackend` | `True` | chDB exposes a direct in-process Arrow buffer (`StreamingResult.record_batch()` returns a native `pa.RecordBatchReader`; `to_arrowTable` uses `res.get_memview()`). `query_df()` is routed through Arrow → pandas with `zero_copy_only=True`. |

### 3.2 Capability negotiation in `Client` (this is where zero-copy is preserved)

```python
# clickhouse_connect/driver/client.py
def query_df(self, sql, *args, **kw):
 if self.backend.supports_zero_copy_arrow:
 tbl = self.backend.query_arrow(sql, ...) # in-process, no socket, no wire format
 return tbl.to_pandas(zero_copy_only=True) # Arrow buffer shared directly with pandas
 return self._native_df_path(sql, ...) # existing HTTP path, behavior unchanged
```

`query_arrow()` / `query_arrow_stream()` themselves are **not** gated by the flag — both backends implement them, and the caller asked for Arrow explicitly. The flag controls only the **routing decision inside `query_df()`** (and analogously `insert_df`): on chDB the Arrow path is strictly better (zero-copy), on HTTP it isn't (no zero-copy benefit, plus the Native parser has better type fidelity for `LowCardinality` / `Enum` / nested / `Decimal`).

For `query()` returning a `QueryResult`, the chDB backend simply emits `Native` format and reuses clickhouse-connect's existing C parser — type fidelity comes first; zero-copy doesn't apply once results are materialized to Python objects anyway.

### 3.3 `.chdb` extension namespace (for chDB-unique APIs)

chDB has capabilities that have no equivalent on a remote ClickHouse server. Rather than polluting the base `Client` interface with optional methods, they are exposed via a capability extension namespace that is **only present when the chdb backend is active**:

```python
client.chdb.query_python(sql, df=my_df) # Python() table function — query in-scope DataFrame, zero-copy
client.chdb.register_function(udf) # register a Python UDF
client.chdb.session_path # in-memory vs persistent file path
client.chdb.cursor() # expose chdb native DB-API cursor (escape hatch)
```

Accessing `client.chdb` on an HTTP backend raises a clear `AttributeError`. The extension object itself ships in **the chdb-io/chdb repo** (`chdb/cc_extension.py`); the `Client` base only provides the mount hook — it does not know what `.chdb` contains.

</details>

<details>
<summary>👤 4. User experience — installation and one-line switch</summary>

```python
import clickhouse_connect

# Default HTTP — existing users: zero changes, full forward compatibility.
client = clickhouse_connect.get_client(host="prod.clickhouse.cloud",
 username="...", password="...")

# Switch to chDB in-process — one line.
client = clickhouse_connect.get_client(backend="chdb", path=":memory:")

# All downstream code is identical across backends.
df = client.query_df("SELECT * FROM events WHERE ts > now() - INTERVAL 1 HOUR")
client.insert_df("events", df)
for row in client.query_row_block_stream("SELECT * FROM huge_table"):
 ...
```

**Installation:**

```bash
pip install clickhouse-connect # existing users, unchanged
pip install clickhouse-connect chdb # users who want chDB add chdb
# or, equivalently, with extras:
pip install 'clickhouse-connect[chdb]' # extras_require declares chdb dependency
```

After `chdb` is installed, its entry-point is auto-discovered and `backend="chdb"` works. The clickhouse-connect package size, dependency graph, and import time are unchanged for users who don't install chdb.

If a user passes `backend="chdb"` without the package installed, `get_client()` raises a clear `BackendNotInstalled` error pointing at the install command. **No runtime auto-install.**

</details>

<details>
<summary>📋 5. Code-change responsibility matrix — who changes what, when</summary>

| Scenario | clickhouse-connect changes? | chdb changes? |
|---|---|---|
| chDB engine upgrade (new version, perf, bug fix) | ❌ no | ✅ `cc_backend.py` updated |
| chDB output-format behavior change (e.g. Arrow schema tweak) | ❌ no | ✅ `cc_backend.py` adapted |
| CH server adds a new public feature → `Client` gains a new API | ✅ add `Client.foo()` (defaults to `raw_query` / `command`) | usually no; optionally implement a fast path in `ChdbBackend` |
| `HttpBackend` perf / connection-pool tweaks | ✅ `HttpBackend` | ❌ no |
| chDB adds a unique capability (new UDF kind, new Python integration) | ❌ no | ✅ extend `.chdb` namespace in `cc_extension.py` |
| Test discovers chDB doesn't support a CH feature | ❌ no | ✅ add a marker to `cc_test_profile.SKIP_MARKERS` |
| Protocol itself needs a new method (e.g. tracing context, new capability) | ✅ add method + default impl + capability flag | optionally implement optimized path |
| User reports buggy SQL behavior on chDB | ❌ no | ✅ chdb team investigates (engine or `cc_backend`) |
| User reports buggy behavior on HTTP | ✅ CH team investigates | ❌ no |

**Invariant: a chDB upstream change never forces a clickhouse-connect change.**

</details>

<details>
<summary>🧪 6. Testing architecture — parametrized fixtures, marker hygiene, skip manifest, CI split, contract suite</summary>

### 6.1 Parametrized fixtures — write once in clickhouse-connect

```python
# clickhouse_connect/tests/integration_tests/conftest.py
import importlib.metadata, os, pytest

def _enabled_backends():
 backends = ["http"] if os.getenv("CH_HOST") else []
 for ep in importlib.metadata.entry_points(group="clickhouse_connect.backends"):
 if ep.name != "http":
 backends.append(ep.name)
 return backends

@pytest.fixture(params=_enabled_backends())
def client(request):
 return _make_client(request.param)
```

**This fixture does not let chdb affect clickhouse-connect's CI** — `entry_points` discovery is driven by what's installed at runtime:

- In the clickhouse-connect repo's CI, **chdb is not installed** (`requirements.txt` / `setup.py` / workflow files don't reference it) → the entry-point group returns empty → `_enabled_backends()` produces only `["http"]` → tests fan out on http only, **behavior identical to today**.
- The same fixture in the chdb-io/chdb repo's CI (which **does** install chdb and clickhouse-connect) automatically picks up the `[chdb]` parameter dimension and runs the full suite against the chdb backend — that's chdb's CI burden, and clickhouse-connect doesn't see it.

So CH developers writing new tests **do not need to know anything about chDB** — their tests fan out across backends for free, and CI behavior is unchanged.

### 6.2 Marker hygiene (the only new habit for CH developers)

Mark tests with the server feature they depend on. This is good test hygiene that pays for itself independently of chDB:

```python
@pytest.mark.requires_distributed
@pytest.mark.requires_keeper
@pytest.mark.requires_replicated
@pytest.mark.requires_grants
@pytest.mark.http_only # transport-specific: compression negotiation, proxy, HTTPS
def test_xxx(client): ...
```

The marker set lives in clickhouse-connect's `pytest.ini` and grows organically.

### 6.3 Skip manifest — owned by chdb

```python
# chdb/cc_test_profile.py ── inside the chdb-io/chdb repo
SKIP_MARKERS = {
 "requires_distributed", "requires_replicated", "requires_keeper",
 "requires_grants", "requires_async_insert", "http_only",
 # ... chdb team maintains this list as engine capabilities evolve
}
SKIP_NODEIDS = {
 "tests/.../test_x.py::test_y": "chdb does not yet support ALTER MODIFY CODEC",
}
```

clickhouse-connect's `conftest.py` loads the profile via entry-point and applies skips dynamically. This code is written **once** and never touched again.

**chDB owns data, not code.** The skip manifest is a declarative list maintained in the chdb repo.

### 6.4 CI matrix split

```
clickhouse-connect repo CI: chdb-io/chdb repo CI:
 matrix: backend=[http] pip install chdb clickhouse-connect
 runs against a real CH server pytest --pyargs clickhouse_connect.tests
 ─ never fails because of chdb ─ runs the entire CH suite on chdb backend
 ─ never pins to a chdb version ─ chdb upstream changes surface here first
 ─ chdb team fixes backend or extends skip list
```

The chdb repo also subscribes to clickhouse-connect's release tags and re-runs the regression on each new CH release. **CH developers are transparent to this pipeline — they do nothing for it.**

### 6.5 Backend-contract suite (clickhouse-connect repo)

A small generic suite asserting invariants any backend must satisfy. Runs against the HTTP backend and a mock backend in CH's CI. The chdb repo runs the same suite against `ChdbBackend`. This is the safety net that guarantees backend behavior is predictable.

</details>

---

## Decoupling mechanisms — summary

```
┌──────────────────────────────────────────────────────────────────┐
│ 1. Backend protocol (code contract) │
│ additive only + default impls + capability flags │
│ → protocol evolution never breaks shipped backends │
├──────────────────────────────────────────────────────────────────┤
│ 2. Entry-point registration (runtime discovery) │
│ clickhouse_connect.backends / .test_profile │
│ → clickhouse-connect does not import chdb; │
│ chdb does not fork clickhouse-connect │
├──────────────────────────────────────────────────────────────────┤
│ 3. .chdb extension namespace (unique-API outlet) │
│ → chDB-only capabilities do not pollute the base interface │
├──────────────────────────────────────────────────────────────────┤
│ 4. Skip profile (testing boundary as data) │
│ markers in CH, data in chdb, application logic in conftest │
│ → CH writes tests without knowing chDB; chdb does not fork │
│ CH's test suite │
├──────────────────────────────────────────────────────────────────┤
│ 5. CI matrix split │
│ CH CI depends on zero chdb state │
│ → chDB upstream changes never turn CH CI red │
└──────────────────────────────────────────────────────────────────┘
```

---

## Next step

If this direction looks right to the maintainers, I'll close #753 and re-submit a PR implementing this design (starting with extracting `HttpBackend` from `HttpClient` as a no-behavior-change refactor, then adding the protocol + registry, then wiring the chdb-side `ChdbBackend` in chdb-io/chdb). Happy to discuss any part of the proposal here or in a sync.

cc @joe-clickhouse @LukeGannon

Backend	`supports_zero_copy_arrow`	Why
`HttpBackend`	`False`	HTTP always crosses wire bytes; the Native parser preserves better type fidelity than an Arrow round-trip would. `query_arrow()` still works (existing HTTP behavior unchanged) — but `query_df()` keeps using the Native parser path.
`ChdbBackend`	`True`	chDB exposes a direct in-process Arrow buffer (`StreamingResult.record_batch()` returns a native `pa.RecordBatchReader`; `to_arrowTable` uses `res.get_memview()`). `query_df()` is routed through Arrow → pandas with `zero_copy_only=True`.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Discussion: pluggable Backend protocol for chDB integration (alternative to #753) #809

TL;DR

Core principle (one sentence)

Before

After

Protocol-evolution discipline

3.1 Backend protocol (defined in clickhouse-connect)

3.2 Capability negotiation in `Client` (this is where zero-copy is preserved)

3.3 `.chdb` extension namespace (for chDB-unique APIs)

6.1 Parametrized fixtures — write once in clickhouse-connect

6.2 Marker hygiene (the only new habit for CH developers)

6.3 Skip manifest — owned by chdb

6.4 CI matrix split

6.5 Backend-contract suite (clickhouse-connect repo)

Decoupling mechanisms — summary

Next step

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Scenario	clickhouse-connect changes?	chdb changes?
chDB engine upgrade (new version, perf, bug fix)	❌ no	✅ `cc_backend.py` updated
chDB output-format behavior change (e.g. Arrow schema tweak)	❌ no	✅ `cc_backend.py` adapted
CH server adds a new public feature → `Client` gains a new API	✅ add `Client.foo()` (defaults to `raw_query` / `command`)	usually no; optionally implement a fast path in `ChdbBackend`
`HttpBackend` perf / connection-pool tweaks	✅ `HttpBackend`	❌ no
chDB adds a unique capability (new UDF kind, new Python integration)	❌ no	✅ extend `.chdb` namespace in `cc_extension.py`
Test discovers chDB doesn't support a CH feature	❌ no	✅ add a marker to `cc_test_profile.SKIP_MARKERS`
Protocol itself needs a new method (e.g. tracing context, new capability)	✅ add method + default impl + capability flag	optionally implement optimized path
User reports buggy SQL behavior on chDB	❌ no	✅ chdb team investigates (engine or `cc_backend`)
User reports buggy behavior on HTTP	✅ CH team investigates	❌ no

Uh oh!

Discussion: pluggable Backend protocol for chDB integration (alternative to #753) #809

Description

TL;DR

Core principle (one sentence)

Before

After

Protocol-evolution discipline

3.1 Backend protocol (defined in clickhouse-connect)

3.2 Capability negotiation in Client (this is where zero-copy is preserved)

3.3 .chdb extension namespace (for chDB-unique APIs)

6.1 Parametrized fixtures — write once in clickhouse-connect

6.2 Marker hygiene (the only new habit for CH developers)

6.3 Skip manifest — owned by chdb

6.4 CI matrix split

6.5 Backend-contract suite (clickhouse-connect repo)

Decoupling mechanisms — summary

Next step

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

3.2 Capability negotiation in `Client` (this is where zero-copy is preserved)

3.3 `.chdb` extension namespace (for chDB-unique APIs)