Skip to content

Fix Parquet writes for multi-record Avro unions - #1250

Merged
freeznet merged 3 commits into
masterfrom
codex/fix-avro-multi-record-union
Aug 28, 2026
Merged

freeznet merged 3 commits into
masterfrom
codex/fix-avro-multi-record-union

Conversation

@freeznet

@freeznet freeznet commented Aug 27, 2026 •

Copy link
Copy Markdown
Member

Problem

Generic Avro conversion always selected the first record branch in a union. Records using a later branch could therefore be converted with an incompatible schema before being written to Parquet.

Changes

  • Resolve the concrete union branch from the native Avro record schema.
  • Preserve the existing fallback for nullable unions with one record branch.
  • Fail explicitly when a multi-record branch cannot be determined.
  • Add a Parquet round-trip regression test for a non-first record branch with an incompatible nested field type.
  • Align the regression test with the repository Checkstyle rules.
  • Update dependency overrides required by the repository vulnerability scan.

Validation

  • JAVA_HOME=/path/to/jdk17 mvn -B -ntp clean license:check checkstyle:check spotbugs:check
  • JAVA_HOME=/path/to/jdk17 mvn -B -ntp -Dtest=ParquetMultiRecordUnionTest test
  • trivy fs --scanners vuln --severity CRITICAL,HIGH --exit-code 1 --timeout 10m .

Copilot AI lite review requested due to automatic review settings August 27, 2026 15:47
@freeznet
freeznet requested a review from a team as a code owner August 27, 2026 15:47

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@github-actions

Copy link
Copy Markdown

@freeznet:Thanks for your contribution. For this PR, do we need to update docs?
(The PR template contains info about doc, which helps others know more about the changes. Can you provide doc-related info in this and future PR descriptions? Thanks)

@github-actions github-actions Bot added the doc-info-missing This pr needs to mark a document option in description label Aug 27, 2026
@freeznet
freeznet merged commit b04f7f2 into master Aug 28, 2026
2 checks passed
@freeznet
freeznet deleted the codex/fix-avro-multi-record-union branch August 28, 2026 03:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

doc-info-missing This pr needs to mark a document option in description

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants