feat: add NamespaceClientTableContext for cached namespace info - #96

Closed

jackye1995 wants to merge 186 commits into

mainfrom

Commits on Mar 9, 2026

chore: ci/cd workflow improvements and fixes (#6127)
esteban
authored
88b2635

Commits on Mar 10, 2026

chore: release beta version 4.0.0-beta.8
Lance Release Bot
committed
e7eb014
fix: replace `fetch_arrow_table` with `to_arrow_table` (#6146)
BubbleCal
authored
31de8d4
perf: parallelize FTS prewarming (#6144)
BubbleCal
authored
0681a08
ci: fix pinned auth action ref in benchmarks workflow (#6153)
Xuanwo
authored
e1aa70a
docs: document the rules for transaction conflicts (#6158)
westonpace
authored
6be1b58

Commits on Mar 11, 2026

fix: handle DataType::Null in adjust_child_validity to prevent panic (#6160)

wjones127
and
claude
authored
3ccd602
fix: persist frag reuse index external file on local filesystem (#6163)

wjones127
and
claude
authored
be5f439
feat: clearer progress reporting for IVF (#6126)
wkalt
authored
fa64837
chore: release beta version 4.0.0-beta.9
Lance Release Bot
committed
e133d82

Commits on Mar 12, 2026

chore: clippy cleanups (#6172)
Xuanwo
authored
31cd8d3
ci: move Linux and Windows jobs to GitHub-hosted runners (#6175)
Xuanwo
authored
98bb073
docs: add alicloud oss configuration (#6167)
FarmerChillax
authored
f3e50d7
fix: avoid empty range reads for zero-length blobs (#6168)
Xuanwo
authored
58f3b84
perf: remove shard content key sorting from distributed merge (#6179)
Xuanwo
authored
f4adbc0
perf(inverted)!: reduce fts indexing time and memory (#6174)

authored
c70ee41
chore: release beta version 4.0.0-beta.10
Lance Release Bot
committed
3f74834

Commits on Mar 13, 2026

docs: update the rules for data replacement conflicts to reflect reality (#6182)
westonpace
authored
46826b1
fix: handle nullable validity layers without def levels (#6187)
Xuanwo
authored
0851b14
fix: preserve merge insert delete-by-source semantics (#6148)
Xuanwo
authored
311ff1a
fix: use to_arrow_reader in benchmark datagen (#6190)
Xuanwo
authored
29075c9
perf(inverted): reuse posting batch builder and merge tail partitions (#6191)
BubbleCal
authored
e675fdb
refactor: rename arrow-scalar to lance-arrow-scalar (#6199)
westonpace
authored
7aa6d33
feat: support vector indices in describe_indices filtering (#6145)

ndpvt-web
and
AI Assistant (Claude)
authored
71c6e21
fix: memory_limit and num_workers params are not passed to index worker (#6197)
BubbleCal
authored
c9c8c46
chore: release beta version 4.0.0-beta.11
Lance Release Bot
committed
e8109ad
fix: disallowing stale credentials from directory namespace (#6194)
hamersaw
authored
f02df43
chore: release beta version 4.0.0-beta.12
Lance Release Bot
committed
1e7b725

Commits on Mar 14, 2026

feat: enable HNSW-accelerated partition assignment for fp16 vectors (#6119)

authored
f0160c7

Commits on Mar 16, 2026

Commits on Mar 17, 2026

Commits on Mar 18, 2026

fix: add missing type hint for producer function (#6133)
Gallardot
authored
fb1566a
fix: maintaining individual fragment operation when calling take_source (#5844)

hamersaw
and
Xuanwo
authored
ae9a2cb
perf: pre-transpose PQ codebook for SIMD-friendly L2 distance (#5923)

wkalt
and
Xuanwo
authored
d59bd23
feat: add file list with sizes to IndexMetadata (#5497)

wjones127
and
claude
authored
023b14b
docs: fix incorrect URLs and cleanup (#5317)
prrao87
authored
28db831
feat: add index segment commit API (#6209)
Xuanwo
authored
c9e5b1e
feat: add abfss:// scheme support for Azure ADLS Gen2 (#6192)
burlacio
authored
1a8b23d
docs: document vector index RAM (training) & storage requirements (#6108)
westonpace
authored
f404d0e

Commits on Mar 19, 2026

feat: support atomic multi-table transactions via namespace manifest (#6173)
XuQianJin-Stars
authored
416f5a6
chore: release beta version 4.0.0-beta.13
Lance Release Bot
committed
60cae4c
docs: remove legacy preview index note (#6218)
Xuanwo
authored
674ccce
chore: bump main to 4.1.0-beta.0
Lance Release Bot
committed
15e16cd

Commits on Mar 20, 2026

Commits on Mar 21, 2026

Commits on Mar 22, 2026

chore: release beta version 4.1.0-beta.1
Lance Release Bot
committed
88c627b

Commits on Mar 23, 2026

fix: adding namespace support to java SDK CommitBuilder from dataset (#6257)
hamersaw
authored
727a45c
chore: release beta version 4.1.0-beta.2
Lance Release Bot
committed
0302473
fix: resolve_latest_location converts errors to not_found unconditionally (#6248)

wkalt
and
claude
authored
baa13ab
perf: batched WAND and new WAND structure, ~50% faster (#6241)
BubbleCal
authored
384fb55
fix: return errors for unsupported fixed-size-list child types (#6253)
myandpr
authored
1d4052c
feat: add OpenDAL Azdls backend for abfss:// with use_opendal flag (#6256)

burlacio
and
Razvan Burlacioiu
authored
40929fe
chore: release beta version 4.1.0-beta.3
Lance Release Bot
committed
244c721

Commits on Mar 24, 2026

Commits on Mar 25, 2026

feat(DirectoryNamespace): support index and transaction related operations (#6196)

zhangyue19921010
and
zhangyue19921010
authored
75f0195
feat: add aimd throttled object store (#6266)
westonpace
authored
e2231fd
feat: support stop-word gaps in phrase queries (#6277)
BubbleCal
authored
d016a82
fix: support hamming distance in IndicesBuilder (#6295)

jmhsieh
and
claude
authored
a6b033c

Commits on Mar 26, 2026

perf: add benchmark for distributed vector merge finalization (#6176)
Xuanwo
authored
305b729
refactor: move DatasetIndexExt out of lance-index (#6280)
Xuanwo
authored
ee324f4
fix: restore namespace build after DatasetIndexExt move (#6302)
Xuanwo
authored
c54f529
feat: actually using namespace error variants (#6275)

hamersaw
and
claude
authored
3a82648
fix: multiple improvements for gh workflows (#6306)

esteban
and
Esteban Gutierrez
authored
2ad415a
feat!: support sampling selected fragments (#6294)
Xuanwo
authored
f0aa55f
feat(java): add non-blocking AsyncScanner with CompletableFuture API (#6102)

beinan
and
claude
authored
7f6fd0e

Commits on Mar 27, 2026

fix(namespace): support nested types in convert_json_arrow_type (#6300)

jiaoew1991
and
claude
authored
8c99eef
feat: add a fast dataset version ID API (#6303)
BubbleCal
authored
08ec995
feat: clarify logical indices and physical index segments (#6270)
Xuanwo
authored
8b8e36a
docs: shorten core major release vote window (#6154)
Xuanwo
authored
05c796c
test: tighten multi-segment vector index coverage and cleanup (#6251)
Xuanwo
authored
9b43852
feat: support non-shared centroid vector index builds (#6296)
Xuanwo
authored
92c45f6
feat(python): add storage_options to IvfModel and PqModel save/load (#6312)
hushengquan
authored
55311f8
perf: optimize stable row_id index build from O(rows) to O(fragments) (#6310)

jiaoew1991
and
claude
authored
36e8b2d
feat: move rate limiting to the object store (#6293)

westonpace
and
claude
authored
209f99b
chore: bump to 5.0.0-beta.1 based on breaking change detection
Lance Release Bot
committed
75b8d2a

Commits on Mar 28, 2026

docs: add conflict handling and FRI guidance (#6304)

westonpace
and
claude
authored
2a4daf1

Commits on Mar 29, 2026

feat: pluggable index cache via CacheBackend trait (#6222)

wjones127
and
claude
authored
7342abf

Commits on Mar 30, 2026

chore: cleanup dead functions and dead_code annotations (#6314)
Xuanwo
authored
277db86
docs(python): prefer uv for local environment setup (#6335)
Xuanwo
authored
38454c8
feat: add write progress callback to InsertBuilder (#6318)

wjones127
and
claude
authored
26bf8f3
feat: support hamming distance in HNSW (#6336)
BubbleCal
authored
30eceac
chore: release beta version 5.0.0-beta.2
Lance Release Bot
committed
34e311c
feat: make MAX_MINIBLOCK_VALUES configurable via env var (#6340)

westonpace
and
claude
authored
ee71812
fix: use StorageOptions::new() in cloud providers to pick up env vars (#6316)

westonpace
and
claude
authored
0c754aa

Commits on Mar 31, 2026

perf: remove O(n²) performance regression in take() with duplicate indices (#6351)

YSBF
and
noreply
authored
2e10283
refactor: remove row-id ordering from vector index merge (#6332)
Xuanwo
authored
da46e7e
fix: avoid full scan for nullable vector fragment sampling (#6341)
Xuanwo
authored
b7f6baa
fix: respect fragment filters during distributed worker training (#6358)
Xuanwo
authored
c869512
feat: expose stable row ids (#6325)

ivscheianu
and
wjones127
authored
507dd6f
feat: change default file format version to 2.1 (#6115)

authored
3c35f48
feat: support distributed IVF_RQ segment builds (#6359)
Xuanwo
authored
0f08160
feat: expose stable row ids (#6325)

ivscheianu
and
wjones127
authored
03dd364
chore(deps): bump pygments from 2.19.2 to 2.20.0 in /python (#6348)
dependabot[bot]
authored
65e2ad0
docs: clarify helper function guidance in AGENTS.md (#6357)
Xuanwo
authored
d4df618
feat: io_uring based file reader (#5777)

westonpace
and
claude
authored
88d8faf

Commits on Apr 1, 2026

feat: expose branch_identifier in python and java bindings (#6360)
majin1102
authored
526a72f
refactor!: align distributed index build around segments (#6313)
Xuanwo
authored
1ef6310
feat: add an arrow-stats crate with the ability to calculate basic stats on arrays (#5967)

westonpace
and
claude
authored
21ae66b
feat: support DataFusion Expr in DeleteBuilder (#6343)

wjones127
and
claude
authored
932d4aa
fix: preserve index details in python index metadata (#6279)
Xuanwo
authored
b08528e
perf: reduce bitmap index build to 1 bitmap in RAM (#6371)

wkalt
and
claude
authored
175a492
feat: support bf16 from pytorch dataset (#6342)

eddyxu
and
claude
authored
21d830a
feat: object store decides scheduler type (#6373)

westonpace
and
claude
authored
1a1094d

Commits on Apr 2, 2026

refactor!: cleanup namespace related APIs (#6186)

authored
d13d7d8
chore: release beta version 5.0.0-beta.3
Lance Release Bot
committed
5504998
feat: support vector query pruning by index segments (#6376)
Xuanwo
authored
402f7b7
feat: support ingest mode for external blob writes (#6356)
Xuanwo
authored
8b59eb6
perf: fix O(N²) version column scan with deletion vectors (#6375)

pengw0048
and
claude
authored
8bd9b90
fix: ensure durable writes actually wait for WAL flush (#6368)

hamersaw
and
claude
authored
2b62004
feat!: add progress monitoring via callbacks for distributed merge (#6210)
vivek-bharathan
authored
cc89488
fix: preserve multipart part ordering in throttle wrapper (#6393)

westonpace
and
claude
authored
a97c647
chore: release beta version 5.0.0-beta.4
Lance Release Bot
committed
d9068e7
chore(deps): bump requests from 2.32.5 to 2.33.0 in /python (#6299)
dependabot[bot]
authored
b36df7d
feat: thread data_size through decode pipeline (#6391)

westonpace
and
claude
authored
36b344f

Commits on Apr 3, 2026

Commits on Apr 4, 2026

feat(namespace): add count_table_rows, insert_into_table, query_table (#6132)

XuQianJin-Stars
and
jackye1995
authored
3c8fe53
chore: release beta version 5.0.0-beta.5
Lance Release Bot
committed
d630106

Commits on Apr 6, 2026

Commits on Apr 7, 2026

fix: avoid corrupting sub-schema merge_insert pages on v2.2 (#6420)

Xuanwo
and
erandagan
authored
4f6e3c1
feat(java): add allowExternalBlobOutsideBases to WriteParams (#6330)

beinan
and
claude
authored
54196a2
fix: skip legacy validation for Overwrite operations (#6418)

hamersaw
and
claude
authored
4e61245
feat: introduce `CacheCodec` for serializing index cache entries (#6223)

wjones127
and
claude
authored
774d1bd

Commits on Apr 8, 2026

docs: add community sync to docs (#6434)
ccmao1130
authored
83ae74b
test: fix vector progress callback timing assertion (#6437)
Xuanwo
authored
87ef5e2
feat: support IVF partitions multi-split (#6423)
BubbleCal
authored
5310f36
feat: add batch_size_bytes to encoding decode stream (#6388)

westonpace
and
claude
authored
40d09bf
feat: add DataFile.create helper for building DataFile metadata (#6427)

westonpace
and
claude
authored
6112a34
feat(java): add Dataset.getZonemapStats() API (#6421)

beinan
and
claude
authored
25a2020
fix: implement next page token support in list namespaces and tables (#6419)
ayao227
authored
943e14a
fix: return error instead of OOM on corrupt binary page offsets (#6392)

westonpace
and
claude
authored
c04c585
fix(namespace): remove trailing empty query in table uri (#6415)
jackye1995
authored
a7aba1f
fix(merge_insert): use sentinel column for NULL-safe source row detection (#6439)

authored
46650e6
ci: add explicit permissions to GitHub Actions workflows (#6440)
wjones127
authored
d503b8e

Commits on Apr 9, 2026

feat(python): add blob-aware to_pandas (#6424)
BubbleCal
authored
05c7ff9
chore: release beta version 5.0.0-beta.6
Lance Release Bot
committed
9b177ad
feat: add configurable FTS index prewarm options (#6298)
BubbleCal
authored
5f4bf81
chore: bump main to 5.1.0-beta.0
Lance Release Bot
committed
7c39a31
fix: warn and clamp LANCE_INITIAL_UPLOAD_SIZE instead of panicking (#6389)
LuciferYang
authored
bec9491
chore: upgrade Rust toolchain to 1.94.0 (#6460)

wjones127
and
claude
authored
f2db129

Commits on Apr 10, 2026

feat(index): support float16 and float64 in IVF_FLAT (#6476)
BubbleCal
authored
a57ec81
chore: release beta version 5.1.0-beta.1
Lance Release Bot
committed
103e947
feat: wire batch_size_bytes to Python and public Rust API (#6428)

westonpace
and
claude
authored
e7369fb
chore: remove custom cast_with_options in favor of upstream (#6344)

wjones127
and
claude
authored
5aca703
feat: clean up transaction files on failed commits (#6319)

wjones127
and
claude
authored
6db7525

Commits on Apr 11, 2026

chore: release beta version 5.1.0-beta.2
Lance Release Bot
committed
df61d95
fix(index): preserve fts prewarm position codec (#6485)
BubbleCal
authored
0634c6a
fix: include column_metadatas and column_infos in CachedFileMetadata::DeepSizeOf (#6480)

jiaoew1991
and
claude
authored
8ac0741
fix: handle FlatBin quantization in optimize_vector_indices_v2 (#6488)
jackye1995
authored
7a65e88
chore: release beta version 5.1.0-beta.3
Lance Release Bot
committed
0a0d53b

Commits on Apr 13, 2026

Commits on Apr 14, 2026

perf: intern RowDatasetVersionMeta inline bytes to reduce manifest memory (#6499)

beinan
and
claude
authored
23526f6
feat(dictionary-namespace): support table related operation (#6308)

zhangyue19921010
and
zhangyue19921010
authored
d37348d
chore: add tracing instrumentation to mem_wal module (#6430)

hamersaw
and
claude
authored
97a89a6
perf: submit I/O requests eagerly in FullZipScheduler (#6513)
hushengquan
authored
9df651b
feat: add configurable blob v2 pack file size (#6508)

hamersaw
and
claude
authored
9672573

Commits on Apr 15, 2026

feat: add NamespaceClientTableContext for cached namespace info across Rust, Python, Java

jackye1995
and
claude
committed
b775da2