Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -55,13 +55,14 @@ jobs:
uses: reviewdog/action-actionlint@83e4ed25b168066ad8f62f5afbb29ebd8641d982 # v1.6.8

test:
name: "Test (Python ${{ matrix.python-version }})"
name: "Test (${{ matrix.os }}, Python ${{ matrix.python-version }})"
needs: lint
runs-on: ubuntu-latest
runs-on: ${{ matrix.os }}
timeout-minutes: 15
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest, macos-latest]
python-version: ["3.10", "3.11", "3.12", "3.13", "3.14", "3.14t"]

steps:
Expand Down Expand Up @@ -95,7 +96,7 @@ jobs:
run: just test-coverage

- name: Upload coverage to Codecov
if: matrix.python-version == '3.10'
if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.10'
uses: codecov/codecov-action@5a1091511ad55cbe89839c7260b706298ca349f7 # v5.5.1
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
Expand Down
2 changes: 1 addition & 1 deletion .vale.ini
Original file line number Diff line number Diff line change
Expand Up @@ -14,5 +14,5 @@ Google.Will = NO
proselint.Annotations = NO

# Internal/temporary files - skip linting
[{.claude/**/*.md,.oaps/**/**.md,tmp/**/*.md}]
[{.claude/**/*.md,.oaps/**/**.md,tmp/**/*.md,PLAN.md}]
BasedOnStyles =
187 changes: 185 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# JSON Lines Table (JSONLT) Python package
# JSONLT Python package

<!-- vale off -->
[![CI](https://github.com/jsonlt/jsonlt-python/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/jsonlt/jsonlt-python/actions/workflows/ci.yml)
Expand All @@ -10,8 +10,191 @@

**jsonlt** is the Python reference implementation of the [JSON Lines Table (JSONLT) specification][jsonlt].

JSONLT is a data format for storing keyed records in append-only files using [JSON Lines](https://jsonlines.org/). The format optimizes for version control diffs and human readability.

> [!NOTE]
> This package is in development and not yet ready for use.
> This package is in development and not yet ready for production use.

## Installation

```bash
pip install jsonlt

# Or

uv add jsonlt
```

## Quick start

### Basic operations

```python
from jsonlt import Table

# Open or create a table with a simple key
table = Table("users.jsonlt", key="id")

# Insert or update records
table.put({"id": "alice", "role": "admin", "email": "alice@example.com"})
table.put({"id": "bob", "role": "user", "email": "bob@example.com"})

# Read a record by key
user = table.get("alice")
print(user) # {"id": "alice", "role": "admin", "email": "alice@example.com"}

# Check if a key exists
if table.has("bob"):
print("Bob exists")

# Delete a record
table.delete("bob")

# Get all records
for record in table.all():
print(record)
```

### Compound keys

JSONLT supports multi-field compound keys:

```python
# Using a tuple of field names for compound keys
orders = Table("orders.jsonlt", key=("customer_id", "order_id"))

orders.put({"customer_id": "alice", "order_id": 1, "total": 99.99})
orders.put({"customer_id": "alice", "order_id": 2, "total": 149.99})

# Access with compound key
order = orders.get(("alice", 1))
```

### Transactions

Use transactions for atomic updates with conflict detection:

```python
from jsonlt import Table, ConflictError

table = Table("counters.jsonlt", key="name")

# Context manager commits on success, aborts on exception
with table.transaction() as tx:
counter = tx.get("visits")
if counter:
tx.put({"name": "visits", "count": counter["count"] + 1})
else:
tx.put({"name": "visits", "count": 1})

# Handle conflicts from concurrent modifications
try:
with table.transaction() as tx:
tx.put({"name": "counter", "value": 42})
except ConflictError as e:
print(f"Conflict on key: {e.key}")
```

### Finding records

```python
from jsonlt import Table

table = Table("products.jsonlt", key="sku")

# Find all records matching a predicate
expensive = table.find(lambda r: r.get("price", 0) > 100)

# Find with limit
top_3 = table.find(lambda r: r.get("in_stock", False), limit=3)

# Find the first matching record
first_match = table.find_one(lambda r: r.get("category") == "electronics")
```

### Table maintenance

```python
from jsonlt import Table

table = Table("data.jsonlt", key="id")

# Compact the table (removes tombstones and superseded records)
table.compact()

# Clear all records (keeps header if present)
table.clear()
```

## API overview

### Table class

The `Table` class is the primary interface for working with JSONLT files.

| Method | Description |
| ------ | ----------- |
| `Table(path, key)` | Open or create a table at the given path |
| `get(key)` | Get a record by key, returns `None` if not found |
| `has(key)` | Check if a key exists |
| `put(record)` | Insert or update a record |
| `delete(key)` | Delete a record, returns whether it existed |
| `all()` | Get all records in key order |
| `keys()` | Get all keys in key order |
| `items()` | Get all (key, record) pairs in key order |
| `count()` | Get the number of records |
| `find(predicate, limit=None)` | Find records matching a predicate |
| `find_one(predicate)` | Find the first matching record |
| `transaction()` | Start a new transaction |
| `compact()` | Compact the table file |
| `clear()` | Remove all records |
| `reload()` | Force reload from disk |

The `Table` class also supports idiomatic Python operations:

- `len(table)` - number of records
- `key in table` - check if key exists
- `for record in table` - iterate over records

### Transaction class

The `Transaction` class provides snapshot isolation and buffered writes.

| Method | Description |
| ------ | ----------- |
| `get(key)` | Get a record from the transaction snapshot |
| `has(key)` | Check if a key exists in the snapshot |
| `put(record)` | Buffer a record for commit |
| `delete(key)` | Buffer a deletion for commit |
| `commit()` | Write buffered changes to disk |
| `abort()` | Discard buffered changes |

### Exception hierarchy

All exceptions inherit from `JSONLTError`:

| Exception | Description |
| --------- | ----------- |
| `ParseError` | Invalid file format or content |
| `InvalidKeyError` | Invalid or missing key |
| `FileError` | File I/O error |
| `LockError` | Cannot obtain file lock |
| `LimitError` | Size limit exceeded |
| `TransactionError` | Transaction state error |
| `ConflictError` | Write-write conflict detected |

### Type exports

For type annotations, the package exports:

- `Key` - A key value (`str`, `int`, or tuple of these)
- `KeySpecifier` - Key field names (single or multiple)
- `JSONObject` - A JSON object (`dict[str, Any]`)
- `Header` - Table header data class

## Documentation

For detailed documentation, tutorials, and the full specification, visit [jsonlt.org/docs](https://jsonlt.org/docs).

## License

Expand Down
6 changes: 4 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ dependencies = ["typing-extensions>=4.15.0"]
dev = [
"basedpyright>=1.36.1",
"codespell>=2.4.1",
"cosmic-ray>=8.4.3",
"cosmic-ray>=8.4.3; sys_platform != 'win32'",
"dirty-equals>=0.11",
"faker>=39.0.0",
"hypothesis>=6.148.7",
Expand All @@ -24,7 +24,7 @@ dev = [
"pytest-codspeed>=4.2.0",
"pytest-cov>=7.0.0",
"pytest-examples>=0.0.18",
"pytest-memray>=1.8.0",
"pytest-memray>=1.8.0; sys_platform != 'win32'",
"pytest-mock>=3.15.1",
"pytest-test-groups>=1.2.1",
"rich>=14.2.0",
Expand Down Expand Up @@ -88,6 +88,8 @@ exclude_lines = [
"if TYPE_CHECKING:",
"@abstractmethod",
"@abc.abstractmethod",
# Platform-specific exclusions
"class _WindowsLock:",
]
precision = 2
show_missing = true
Expand Down
35 changes: 34 additions & 1 deletion src/jsonlt/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,41 @@

from importlib.metadata import version

from ._exceptions import (
ConflictError,
FileError,
InvalidKeyError,
JSONLTError,
LimitError,
LockError,
ParseError,
TransactionError,
)
from ._header import Header
from ._json import JSONArray, JSONObject, JSONPrimitive, JSONValue
from ._keys import Key, KeySpecifier
from ._table import Table
from ._transaction import Transaction

__version__ = version("jsonlt")

__all__ = ["Table", "__version__"]
__all__ = [
"ConflictError",
"FileError",
"Header",
"InvalidKeyError",
"JSONArray",
"JSONLTError",
"JSONObject",
"JSONPrimitive",
"JSONValue",
"Key",
"KeySpecifier",
"LimitError",
"LockError",
"ParseError",
"Table",
"Transaction",
"TransactionError",
"__version__",
]
55 changes: 55 additions & 0 deletions src/jsonlt/_constants.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
"""Constants defining implementation limits and protocol version.

These constants define the minimum required limits per the JSONLT specification.
Implementations may support larger values.
"""

from typing import Final

# JSONLT specification version
JSONLT_VERSION: Final[int] = 1
"""The JSONLT specification version implemented."""

# Minimum required limits per specification
MAX_KEY_LENGTH: Final[int] = 1024
"""Maximum supported key length in bytes.

The key length is the number of bytes in the key's JSON representation
when encoded as UTF-8. For example, "alice" is 7 bytes (including quotes).
"""

MAX_RECORD_SIZE: Final[int] = 1_048_576
"""Maximum supported record size in bytes (1 MiB).

The record size is the number of bytes in the record's JSON serialization
using deterministic serialization, encoded as UTF-8.
"""

MIN_NESTING_DEPTH: Final[int] = 64
"""Minimum supported JSON nesting depth.

Nesting depth is the maximum number of nested JSON objects and arrays
at any point within a value, where the outermost value is at depth 1.
"""

MAX_TUPLE_ELEMENTS: Final[int] = 16
"""Maximum number of elements in a tuple key.

Tuple keys may contain at most 16 elements. Key specifiers with more
than 16 field names are invalid.
"""

# Valid integer key range (IEEE 754 double-precision safe integers)
MAX_INTEGER_KEY: Final[int] = 2**53 - 1
"""Maximum valid integer key value (9007199254740991).

This is the maximum integer that IEEE 754 double-precision floating-point
can represent exactly, ensuring interoperability across languages.
"""

MIN_INTEGER_KEY: Final[int] = -(2**53) + 1
"""Minimum valid integer key value (-9007199254740991).

This is the minimum integer that IEEE 754 double-precision floating-point
can represent exactly, ensuring interoperability across languages.
"""
Loading