Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 17 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,22 @@ with zudb.connect("social.zu1", read_only=True) as conn:

Edges name rows by position, counting from zero, because at load time a row has no other name. Columns may hold booleans, integers, floats, strings, dates, times, datetimes or durations, one kind to a column, and the GIL is released for the write.

## Adding to one that exists

`load` writes a database and an appender grows one. Rows go into per-column buffers in memory and a flush turns the whole buffer into one commit, so a million rows cost one commit rather than a million.

```python
with conn.appender("person") as rows:
for uid, name in enumerate(names):
rows.append_row([uid, name])
```

A row is every column of the table, in the order the table declares them, and the columns come from the table rather than from the first row, so a value that does not belong in one is refused by the call that appended it and the message names the column. A rel table takes rows too, and a row of one is the two ends of an edge as offsets into the tables it runs between, which is how a Python program adds an edge at all.

The `with` block closes, and closing flushes, including on the way out of a block that raised: a load that stopped partway is better served by its rows arriving than by them vanishing, and `discard()` is there for the caller who wants the other answer. A flush that fails keeps its rows, so what did not go in is still there to look at.

On this machine 200,000 rows of an integer and a string take 11 ms to buffer through `append_rows` and 93 ms to flush, which is 1.9 million rows a second including the commit. The same rows one call at a time cost 32 ms of buffering, since a call is a call. Against `INSERT`, 2,000 rows take 32 seconds a row at a time and 28 ms through an appender, and the gap widens with the table because every `INSERT` is a commit and a fold.

## Reading a result as columns

A result is rows to iterate and columns to hand to something else. The columns go out over the Arrow C Data Interface, so pyarrow, pandas and polars each read the same buffers and none of them gets a Python object per cell.
Expand All @@ -77,7 +93,7 @@ The stub is checked against the module it describes in CI: griffe reads the stub

## What works today

The list above is what this client is for. What it does so far is the core of it: `connect`, `execute` and `sql` with named parameters, results that iterate and fetch, values as Python objects both ways including dates, times, datetimes and durations, `Node`, `Rel` and `Path` as classes, `load` for building a graph with edges in it, every condition as an exception class carrying its code, its position and its documentation link, results as Arrow columns and as pandas and polars frames, stubs inside the wheel with a gate that keeps them true, and the GIL released around every statement, every load and every copy out. `register` and the interrupt are next, and each one lands with the tests that say it works.
The list above is what this client is for. What it does so far is the core of it: `connect`, `execute` and `sql` with named parameters, results that iterate and fetch, values as Python objects both ways including dates, times, datetimes and durations, `Node`, `Rel` and `Path` as classes, `load` for building a graph with edges in it, an appender for growing one, every condition as an exception class carrying its code, its position and its documentation link, results as Arrow columns and as pandas and polars frames, stubs inside the wheel with a gate that keeps them true, and the GIL released around every statement, every load and every copy out. `register` and the interrupt are next, and each one lands with the tests that say it works.

## Wheels

Expand Down
2 changes: 2 additions & 0 deletions python/zudb/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
from __future__ import annotations

from ._zudb import (
Appender,
Connection,
Duration,
Node,
Expand Down Expand Up @@ -43,6 +44,7 @@
"connect",
"load",
"Connection",
"Appender",
"Result",
"Node",
"Rel",
Expand Down
41 changes: 41 additions & 0 deletions python/zudb/_zudb.pyi
Original file line number Diff line number Diff line change
Expand Up @@ -69,13 +69,54 @@ class Connection:
def sql(self, statement: str, params: Mapping[str, Value] | None = None) -> Result:
"""The same call, named for the way it reads in a notebook."""

def appender(self, table: str) -> Appender:
"""Opens an appender on `table`, for loading rows into a database that already exists."""

def close(self) -> None:
"""Closes the connection and frees what it held."""

def __enter__(self) -> Connection: ...
def __exit__(self, *_exception: object) -> bool: ...
def __repr__(self) -> str: ...

class Appender:
"""Rows on their way into a table, buffered until they are flushed."""

@property
def table(self) -> str:
"""The table this appender writes to."""

@property
def buffered(self) -> int:
"""Rows buffered and not yet written."""

@property
def committed(self) -> int:
"""Rows this appender has committed, across every flush."""

@property
def closed(self) -> bool:
"""Whether this appender has been closed."""

def append_row(self, row: Iterable[Value]) -> None:
"""Appends one row, which is a sequence of one value per column of the table."""

def append_rows(self, rows: Iterable[Iterable[Value]]) -> int:
"""Appends every row of an iterable of rows."""

def flush(self) -> int:
"""Writes every buffered row and makes it readable."""

def discard(self) -> int:
"""Throws away what is buffered and answers how many rows that was."""

def close(self) -> int:
"""Flushes what is left and answers how many rows this appender committed in all."""

def __enter__(self) -> Appender: ...
def __exit__(self, *_exception: object) -> bool: ...
def __repr__(self) -> str: ...

class Result:
"""The rows a statement gave back."""

Expand Down
Loading
Loading