Skip to content

go/writer: add variant parquet type#4487

Draft
jacobmarble wants to merge 2 commits into
mainfrom
jgm-parquet-variant
Draft

go/writer: add variant parquet type#4487
jacobmarble wants to merge 2 commits into
mainfrom
jgm-parquet-variant

Conversation

@jacobmarble
Copy link
Copy Markdown
Contributor

@jacobmarble jacobmarble commented May 15, 2026

Description:

Adds LogicalTypeVariant to the go/writer parquet package. Variant is a group of two required BYTE_ARRAY children (value and metadata) per the Parquet Variant spec, enabling connectors to emit JSON objects/arrays as native Variant columns instead of raw strings.

New options WithParquetSchemaObjectAsVariant() and WithParquetSchemaArrayAsVariant() let connectors opt in. A getVariantVal helper converts common Go types to variant.Value at write time.

Notes for reviewers:

  • The first commit (a77f7de go/writer: column-oriented parquet write buffer) has its own PR at go/writer: column-oriented parquet write buffer #4485 — please review/merge that first. This PR contains only the top commit.
  • LogicalTypeVariant maps to two physical parquet columns, so physicalColumnCount() was added to ParquetSchemaElement and columnChunkCount accounting was updated accordingly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant