Physical-plan serialization drops ProjectionExec metadata

Physical-plan serialization

Before ProjectionExecNode serialized:

PhysicalPlanNode input = 1;
repeated PhysicalExprNode expr = 2;
repeated string expr_name = 3;

On decode, DataFusion reconstructed the projection which is fine when the complete output schema can be derived from the expression and input but metadata cannot be.

Given:

Before serialization:

Input field:
  value: Int32
  metadata: {}

Projection expression:
  value@0 AS value

ProjectionExec output schema:
  value: Int32
  field metadata: {"field-key": "field-value"}
  schema metadata: {"schema-key": "schema-value"}

---

After serialization and decoding:

ProjectionExec output schema:
  value: Int32
  field metadata: {}
  schema metadata: {}

this was roginally found in #24670 but not fixed there since it was out of optimizer scope / unrelated bug

I have verified this is not a regression from 54.1.0 -> 55.0.0 since serialization in 54 did not support metadata. This issue was exposed by the release of 55.0.0 during testing of the related issue but it is not a regression. I think resolving this will require a change at the protobuf definition which will be outside of the patch release. I do think this should be fixed, but not back ported to 55.1.0.

@timsaucer

@timsaucer I implemented the metadata-preserving serialization change in #25009 . It adds an optional schema only for metadata overrides and restores it through the existing metadata-only constructor. Let me know if this looks good! thanks.

@goutamadwant

hey @goutamadwant I took a look at the PR and left some comments. Just checking in on progress on this guy 👍

fix: preserve ProjectionExec metadata during serialization (apache#25009

Navigation Menu

Repository navigation

Physical-plan serialization drops ProjectionExec metadata #24695