Skip to content

Support blob v2 source columns in ADD COLUMNS FROM #775

Description

@hfutatzhanghb

Context

Follow-up from #764 (comment).

Current behavior

ALTER TABLE ... ADD COLUMNS ... FROM ... does not accept a blob v2 source column when the target configures <column>.lance.encoding = blob.

Spark reads a blob v2 column as a descriptor struct. AddColumnsBackfillExec passes the query schema through SchemaConverter.processSchemaWithProperties before AppendData is built. Because the source field is a struct rather than Spark BINARY, addBlobMetadata rejects it with:

Blob column '<name>' must have BINARY type

The rejection happens before LanceBlobV2CopyThroughRule can rewrite descriptor reads into blob copy references. By contrast, direct INSERT INTO ... SELECT between blob v2 tables supports descriptor-to-blob copies and is covered by BaseBlobV2CopyTest.

Question

Should ADD COLUMNS ... FROM also support direct blob v2 source columns, analogous to INSERT INTO ... SELECT?

Possible direction

  • Preserve or expose blob v2 field metadata during AddColumnsBackfillExec schema processing.
  • Let the command reach an AppendData plan that LanceBlobV2CopyThroughRule can rewrite, or add an equivalent copy-through path for this command.
  • Keep the requirement that _rowaddr and _fragid address target rows.
  • Add integration tests for copying blob bytes from a blob v2 source table into a newly added blob v2 target column.

Current workaround

Copy blob columns between existing tables using direct INSERT INTO ... SELECT, as documented in docs/src/operations/dml/insert-into.md.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions