Skip to content

feat(perf): avoid materializing the input table if given a duckdb-backed ibis.Table#73

Open
NickCrews wants to merge 1 commit intomanzt:mainfrom
NickCrews:ibis-table-perf
Open

feat(perf): avoid materializing the input table if given a duckdb-backed ibis.Table#73
NickCrews wants to merge 1 commit intomanzt:mainfrom
NickCrews:ibis-table-perf

Conversation

@NickCrews
Copy link
Copy Markdown
Contributor

@NickCrews NickCrews commented May 30, 2025

A smaller version of #72

Tested with

import sqlite3
from pathlib import Path

import ibis

import quak

conn = ibis.duckdb.connect()

memtable = ibis.memtable({"x": [1, 2, 3], "y": [4, None, 6], "z": ["a", "b", "c"]})
physical_table = conn.create_table("physical_table", memtable)

sqlite_path = Path("test.sqlite")
if not sqlite_path.exists():
    sqlite_conn = sqlite3.connect(sqlite_path)
    sqlite_conn.execute(
        "CREATE TABLE IF NOT EXISTS test (x INTEGER, y INTEGER, z TEXT)"
    )
    sqlite_conn.execute("INSERT INTO test (x, y, z) VALUES (1, 4, 'a')")
    sqlite_conn.execute("INSERT INTO test (x, y, z) VALUES (2, NULL, 'b')")
    sqlite_conn.execute("INSERT INTO test (x, y, z) VALUES (3, 6, 'c')")
    sqlite_conn.commit()

table_backed_by_sqlite = conn.read_sqlite(sqlite_path, table_name="test")

quak.Widget(memtable)
quak.Widget(physical_table)
quak.Widget(table_backed_by_sqlite)

@NickCrews
Copy link
Copy Markdown
Contributor Author

@manzt if I fix up the merge conflicts, does this look like the right idea and can it get merged?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant