build: bump lance.version to 12.0.0-rc.1

[hamersaw]

Summary

Bumps lance.version from 11.0.0-beta.21 to 12.0.0-rc.1 and adapts the connector to the API and behaviour changes that come with it.

lance-core 12 declares lance-namespace 0.11.1 and bundles namespace implementations compiled against it, so lance-namespace.version has to move from 0.8.6 to 0.11.1 at the same time — leaving the pin at 0.8.6 fails to resolve org.lance.namespace.model classes.

API changes

Change	Reason
queryTable(...) → .getData()	LanceNamespace.queryTable now returns QueryTableResponse rather than byte[]
Namespace test doubles	namespaceExists/tableExists return response objects rather than void
RewriteRows update mode in both SparkPositionDeltaWrite copies	Update operations must set updateMode explicitly — the transaction format cannot persist an absent mode distinctly from RewriteRows, so this preserves what absence previously implied
Widened storage version assertions	STABLE now resolves to file version 2.2
Null-struct test asserts failure	File version 2.0 now rejects null structs instead of silently reading them back as non-null structs with null children

The updateMode change alone accounts for 58 of the 70 test failures the bump initially caused.

Blob v2

Lance 12 resolves both the default and stable file format versions to 2.2, and 2.2 rejects the legacy lance-encoding:blob encoding outright.

BlobUtils.fileFormatSupportsBlobV2 treated an unset version and named aliases as unable to store blob v2, so those tables were written with legacy blob metadata and then failed at creation. It now treats unset and stable as blob v2 capable, matching what Lance resolves them to. An explicit version below 2.2 still selects the legacy encoding, which remains valid there, and legacy maps to 0.1 and still returns false.

This changes the default blob encoding for tables created without an explicit file_format_version: they now get blob v2 and read back as descriptor structs rather than raw BINARY. Tables that need the v1 encoding must pin a version below 2.2. docs/src/config.md is updated to match.

The blob v1 suites created their tables without a version, so they now pin 2.1 to go on exercising the v1 path, including the __blob_size and __blob_pos virtual columns that blob v2 replaces with descriptor struct fields.

Test status

./mvnw test -pl lance-spark-3.5_2.13 -am → 1397 tests, 0 failures, 0 errors.

The same classes are green on 11.0.0-beta.21, and lance-spark-3.4_2.12, lance-spark-4.0_2.13 and lance-spark-4.2_2.13 all test-compile.

Why this is still a draft

org.lance:lance-core:12.0.0-rc.1 is not published to Maven Central. The v12.0.0-rc.1 Java publish workflow has not completed — the tag is several hours old and Central's newest lance-core is still 12.0.0-beta.18. CI cannot resolve the dependency until that lands, and it may be worth checking whether that release job is stuck.

Everything above was therefore verified against 12.0.0-beta.18, which shares the v12 API surface. The pin needs re-verifying against the real rc.1 artifact once it publishes.

🤖 Generated with Claude Code

https://claude.ai/code/session_017acNiGMFxRXWVGnpkFFdhK

[lance-gatekeeper]

⚠️ Gate recommendation: approve with a non-blocking risk.

The v12 API, namespace, transaction, and default/stable blob changes are aligned and exercised across the supported build matrix. The valid next file-format alias still selects legacy blob metadata, so blob table creation fails; users can pin stable or 2.3 as a mitigation.

Please mark this PR with the breaking-change label.

[lance-gatekeeper]

next is also a valid Lance selector and resolves to file format 2.3, but this branch returns false for it. That selects lance-encoding:blob and CREATE TABLE fails instead of creating blob v2. Recognizing next here (or delegating alias resolution when lance-core exposes it) makes aliases follow the same ≥2.2 contract. Users can currently recover by pinning stable or 2.3, so this is non-blocking.

Reproducer

Against this head, run:

spark.sql(
    "CREATE TABLE lance_ns.default.blob_next (id INT, data BINARY) USING lance "
        + "TBLPROPERTIES ('data.lance.encoding' = 'blob', "
        + "'file_format_version' = 'next')");

Expected: the table is created with lance.blob.v2 metadata.

Observed: IllegalArgumentException: Legacy blob columns ... are not supported for file version >= 2.2.

[FANNG1]

#819 overlaps this almost exactly — same adaptations, pinned to 12.0.0-beta.18. Happy either way; I'll close or rebase mine to suit.

12.0.0-rc.1 is on Maven Central now, so the dependency should resolve — worth re-triggering CI.

The one real difference is blob encoding when no version is pinned. We agree on stable → v2; we differ on unset: this PR gives blob v2, #819 pins the table to 2.1 and keeps v1. That is user-visible — those columns read back as descriptor structs instead of BINARY and __blob_size/__blob_pos disappear, which is why the v1 suites needed pinning here. Both defensible, just worth deciding deliberately.

FWIW #819 is green on beta.18 including all seven integration jobs. test_create_btree_index_on_int flakes on external-sort memory — it does that on lance 11 too, and passes on re-run.