Context
Follow-up from #764 (comment).
Current behavior
ALTER TABLE ... ADD COLUMNS ... FROM ... does not accept a blob v2 source column when the target configures <column>.lance.encoding = blob.
Spark reads a blob v2 column as a descriptor struct. AddColumnsBackfillExec passes the query schema through SchemaConverter.processSchemaWithProperties before AppendData is built. Because the source field is a struct rather than Spark BINARY, addBlobMetadata rejects it with:
Blob column '<name>' must have BINARY type
The rejection happens before LanceBlobV2CopyThroughRule can rewrite descriptor reads into blob copy references. By contrast, direct INSERT INTO ... SELECT between blob v2 tables supports descriptor-to-blob copies and is covered by BaseBlobV2CopyTest.
Question
Should ADD COLUMNS ... FROM also support direct blob v2 source columns, analogous to INSERT INTO ... SELECT?
Possible direction
- Preserve or expose blob v2 field metadata during
AddColumnsBackfillExec schema processing.
- Let the command reach an
AppendData plan that LanceBlobV2CopyThroughRule can rewrite, or add an equivalent copy-through path for this command.
- Keep the requirement that
_rowaddr and _fragid address target rows.
- Add integration tests for copying blob bytes from a blob v2 source table into a newly added blob v2 target column.
Current workaround
Copy blob columns between existing tables using direct INSERT INTO ... SELECT, as documented in docs/src/operations/dml/insert-into.md.
Context
Follow-up from #764 (comment).
Current behavior
ALTER TABLE ... ADD COLUMNS ... FROM ...does not accept a blob v2 source column when the target configures<column>.lance.encoding = blob.Spark reads a blob v2 column as a descriptor struct.
AddColumnsBackfillExecpasses the query schema throughSchemaConverter.processSchemaWithPropertiesbeforeAppendDatais built. Because the source field is a struct rather than SparkBINARY,addBlobMetadatarejects it with:The rejection happens before
LanceBlobV2CopyThroughRulecan rewrite descriptor reads into blob copy references. By contrast, directINSERT INTO ... SELECTbetween blob v2 tables supports descriptor-to-blob copies and is covered byBaseBlobV2CopyTest.Question
Should
ADD COLUMNS ... FROMalso support direct blob v2 source columns, analogous toINSERT INTO ... SELECT?Possible direction
AddColumnsBackfillExecschema processing.AppendDataplan thatLanceBlobV2CopyThroughRulecan rewrite, or add an equivalent copy-through path for this command._rowaddrand_fragidaddress target rows.Current workaround
Copy blob columns between existing tables using direct
INSERT INTO ... SELECT, as documented indocs/src/operations/dml/insert-into.md.