Skip to content

Add support for named Avro schema references when translating Avro schemas to Arrow schemas - #14

Merged
Tyler-Sch merged 1 commit into
Tyler-Sch:mainfrom
suket123:schema_ref
Sep 13, 2026
Merged

Tyler-Sch merged 1 commit into
Tyler-Sch:mainfrom
suket123:schema_ref

Conversation

@suket123

@suket123 suket123 commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

Support named Avro schema references

Summary

Add support for named Avro schema references when translating Avro schemas to Arrow schemas.

Problem

Valid Avro schemas can define a named record once and reuse it in later fields. The Avro parser represents these uses as Schema::Ref, which previously reached an unimplemented branch and caused a panic during deserialization.

Changes

  • Resolve named schemas using Apache Avro's resolved-schema name map.
  • Expand named references while preserving the containing Arrow field name and properties.
  • Detect recursive references and return an explanatory error because recursive types cannot be represented directly by the generated Arrow schema.
  • Add coverage for Arrow schema translation, datum deserialization, and recursive-reference errors.

Testing

cargo test --workspace

All workspace tests and documentation tests pass.

Risk

Low. The change only affects schemas containing named references; existing schema translations continue through their original paths.

Breaking Changes

None.

@Tyler-Sch Tyler-Sch left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR.

Overall this looks pretty good and I can merge it after it builds.

A few things to note:

  • the PR makes schema refs work on deserialize only. A round trip with serialize would fail
  • The repo has a fast path (decodes straight into Arrow builders, fast_decode.rs / fast_encode.rs) and a slow path (goes through apache_avro::types::Value). Schemas with refs will always take the slow path since is_supported doesn't recognize Schema::Ref.

Otherwise, things look good. Thanks for the contribution.

@Tyler-Sch
Tyler-Sch merged commit 0447e7a into Tyler-Sch:main Sep 13, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants