Skip to content

Add Lance namespace read/write support#35

Draft
FANNG1 wants to merge 2 commits into
daft-engine:mainfrom
FANNG1:namespace
Draft

Add Lance namespace read/write support#35
FANNG1 wants to merge 2 commits into
daft-engine:mainfrom
FANNG1:namespace

Conversation

@FANNG1

@FANNG1 FANNG1 commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add a shared Lance namespace resolver for URI-or-namespace validation, namespace client construction, table resolution, and storage option merging
  • wire namespace table support through daft.read_lance, DataFrame.write_lance, LanceDataSink, and worker scan reconstruction
  • add dir namespace coverage plus a gated Gravitino Lance REST namespace integration test

Validation

  • make check-format
  • .venv/bin/python -m py_compile daft_lance/namespace.py daft_lance/_lance.py daft_lance/lance_data_sink.py daft_lance/lance_scan.py daft_lance/utils.py tests/io/lancedb/test_namespace_rest_e2e.py
  • .venv/bin/python -m pytest tests/io/lancedb/test_namespace.py tests/io/lancedb/test_namespace_rest_e2e.py -v
  • DAFT_LANCE_REST_URI=http://127.0.0.1:9101/lance NO_PROXY='' no_proxy='' AWS_ACCESS_KEY_ID=minioadmin AWS_SECRET_ACCESS_KEY=minioadmin AWS_ENDPOINT_URL=http://127.0.0.1:9000 AWS_REGION=us-east-1 AWS_DEFAULT_REGION=us-east-1 AWS_ALLOW_HTTP=true .venv/bin/python -m pytest tests/io/lancedb/test_namespace_rest_e2e.py -v

@FANNG1 FANNG1 marked this pull request as draft June 19, 2026 12:17
…tion

Build on the initial namespace read/write support:

- Extend namespace params (namespace_impl/namespace_properties/table_id)
  to merge_columns, merge_columns_df, create_scalar_index, compact_files,
  threading namespace kwargs into their commit sites.
- Add an explicit daft_lance.write_lance() and make the daft monkeypatch
  opt-in via patch_daft(); wire write_lance(mode="merge") through namespace.
- Fix _declare_table fallback that always raised ImportError on the
  CreateEmptyTableRequest import; keep only declare_table.
- Narrow overwrite resolution to only declare on TableNotFound instead of
  swallowing every describe_table exception.
- Drop dead code (open_lance_dataset, pylance<5 compat) and dedupe the
  worker dataset-reopen logic into open_dataset_from_open_kwargs.
- Cover all entry points with dir-namespace tests plus a Gravitino REST
  namespace e2e roundtrip.
- Work around an upstream daft+lance native teardown SIGSEGV (unrelated to
  test outcomes) via a pytest_unconfigure hard-exit after reporting.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant