feat: allow extension PhysicalExprs to decode via a per-type registry

[adriangb]

Which issue does this PR close?

• Closes Allow extension PhysicalExprs to decode via a per-type registry instead of PhysicalExtensionCodec #24626.

Rationale for this change

The user-visible problem

A library that ships its own PhysicalExpr has to write a PhysicalExtensionCodec to serialize it, in a file separate from the expression, and two such libraries on one session have to compose their codecs by hand. PhysicalExtensionExprNode carries an opaque payload and the encoded children but no type discriminator, so the codec is the discriminator: ComposedPhysicalExtensionCodec writes the position of the encoding codec into the payload bytes, the decoding side must register the same codecs in the same order, and a name collision between two independent crates cannot be detected.

A decoder also had no way to reach session state. PhysicalExprDecodeCtx had schema() and decode() but nothing like ExecutionPlanDecodeCtx::task_ctx(), so an extension expression could not resolve a UDF from the function registry or read session configuration while decoding.

Where this sits in the bigger picture

Physical-plan serialization is moving from central downcast_ref chains in datafusion-proto to per-type hooks that live next to each type: try_to_proto (a trait method) plus an inherent try_from_proto (a constructor). #22418 did this for built-in PhysicalExprs and #23494 for ExecutionPlans. Third-party types could use the encode half but not the decode half. This PR closes the loop for PhysicalExpr; #24631 does the same for ExecutionPlan with the same shape and the same names (*FromProto for the decode contract, Extension*FromProto pairing the wire name with the constructor, register_*/decode_* facades over one shared registry, attached through SessionConfig::with_extension).

This PR is stacked on #24631, which adds the shared ProtoDecoderRegistry that both kinds register into. Review the commits after that PR's head; it must land first.

How this relates to PhysicalExtensionCodec

The codec is not deprecated and this PR does not touch ComposedPhysicalExtensionCodec. The decode rule is:

• A PhysicalExtensionExprNode with an expr_name the decoding session has registered decodes through the expression's own try_from_proto. A failure there is fatal: falling through to the codec would let it wrongly decode a payload it never wrote.
• Anything else (no name, or a name this session does not know) takes PhysicalExtensionCodec::try_decode_expr exactly as before. If the codec also fails, the error names the missing registration and lists what the session does know.

The codec keeps UDF/UDAF/UDWF payloads (a function is an instance, not a type), anything not yet migrated, and the released FFI_PhysicalExtensionCodec that datafusion-python exposes. ComposedPhysicalExtensionCodec is the part that becomes deprecable once plans and expressions are registry-backed; that is a separate, later decision.

How FFI will work

The registry stores decoders as a private fn pointer; the public surface is ExtensionExprFromProto, extension_expr_node and the three facade functions. The storage can later grow to carry an FFI vtable and private data, and a lower-level entry point that takes a name and a decoder object can be added, without breaking anything shipped here. PhysicalExprNode crosses an ABI as prost bytes, the pattern datafusion-ffi already uses for Statistics.

What changes are included in this PR?

Four stacked commits, each green on its own, on top of #24631. Built-in expression serialization is untouched: they keep their inherent try_from_proto, dispatched from their own ExprType variant.

1. fix: stamp expr_id on nodes returned by PhysicalExpr::try_to_proto. serialize_physical_expr_with_converter computes the expression's identity but returned the node from a try_to_proto hook verbatim — and every built-in writes expr_id: None. Deduplication under DeduplicatingProtoConverter therefore only worked for expressions that stamped their own id (DynamicFilterPhysicalExpr). The driver now stamps it for everyone; no hook has to know about it.
2. refactor: make PhysicalExprDecodeCtx generic over its session. See the rationale above. PhysicalExprDecodeCtx<'a, S> holds its session by value; datafusion_physical_expr::proto::ExprDecodeSession (a &dyn FunctionRegistry plus a &ConfigOptions, reachable as function_registry() and config_options()) is what the built-ins decode under, built by datafusion-proto from its TaskContext. PhysicalSortExpr::try_from_proto and the ordering helpers, which need no session, stay where they are, generic over S. No crate gains a dependency. PhysicalExprDecodeCtx::new takes the session as a third argument.
3. feat: add an optional expr_name discriminator to PhysicalExtensionExprNode. optional string expr_name = 3; — additive and proto3-compatible. Nothing reads it yet.
4. feat: decode extension PhysicalExprs through a session-scoped registry.
   • An extension expression implements one trait, ExtensionExprFromProto, which pairs the name it is dispatched by with the constructor that rebuilds it. Its try_to_proto is one call — extension_expr_node::<Self>(ctx, payload, self.children()) — which builds the PhysicalExtensionExprNode and stamps Self::NAME, so the encoded name and the registry key cannot drift apart. Built-ins do not implement it, so register_physical_expr::<Column>(..) does not compile and no built-in code or caller changes at all.
   • extension_expr_node is a free function rather than a method on PhysicalExprEncodeCtx, because that context lives in datafusion-physical-expr-common, below the crate that can name ExtensionExprFromProto — the same layering that makes the decode context generic over its session. Its try_from_proto receives the whole PhysicalExprNode, matches the Extension variant, and decodes inputs with ctx.decode_children_expressions; ctx.session() gives it the function registry and the options.
   • Registration goes into the ProtoDecoderRegistry added by feat: allow extension ExecutionPlans to decode via a per-type registry instead of PhysicalExtensionCodec #24631: build one, call register_physical_expr::<T>(&mut registry) for every extension expression the session must decode, and attach it with the existing SessionConfig::with_extension. Registering the same type twice is a no-op (keyed by TypeId); a different type under a taken name is an error. The registry is keyed by the trait as well as the name, so a plan and an expression may use the same name, and one session carries one registry rather than one per kind. This crate contributes only the dyn PhysicalExpr face of it: register_physical_expr, decode_physical_expr and physical_expr_names.
   • decode_physical_expr reads the name off the node rather than taking it as an argument, so a caller cannot pair a node with a name it does not carry. When the codec fallback also fails, the missing registration is added as context on the codec's own error, so the kind the codec chose still reaches a caller that matches on it.
   • datafusion::physical_plan::proto re-exports ExtensionExprFromProto, ProtoDecoderRegistry, the three facade functions, ExprDecodeSession and both contexts, so an extension crate can import everything it needs from the same module the ExecutionPlan facade lives in.
   • Decode dispatch in datafusion-proto, per the rule above, plus the upgrade-guide entry (in 56.0.0).

Why a separate trait rather than a method on PhysicalExpr

Three things rule out putting the decoder on PhysicalExpr next to try_to_proto, each verified in-tree rather than reasoned about:

1. An associated constant makes a trait not dyn-compatible (E0038), which breaks every Arc<dyn PhysicalExpr>. So the wire name can never live on PhysicalExpr.
2. Cargo unifies features across the graph, so a required method on a proto-gated trait is E0046 in a downstream crate that never enabled the feature. That is why try_to_proto returns Ok(None) by default, and that invariant is load-bearing for anything else added there.
3. With a default, a forgotten decoder becomes a decode-time NotImplemented instead of a compile error.

A separate trait avoids all three, and once the built-ins implement it too it is not a second mechanism but the formalization of the one that already existed. The encode/decode asymmetry is structural, and try_to_proto's docs now say so: encoding has a receiver and is dispatched through &dyn PhysicalExpr; decoding is a constructor with no self and can never be reached that way.

Session scoping goes through SessionConfig extensions rather than a new field on TaskContext/SessionState. A real field would force datafusion-physical-expr-common/proto on for everyone; datafusion-execution takes that dependency with default-features = false precisely so crates that never serialize pay nothing.

What is the testing strategy for this PR?

datafusion/proto/tests/cases/plans/expr_registry.rs, 12 tests around a TaggedExpr<K> that decodes through the registry; K is a zero-sized marker carrying the wire name (and whether the decoder fails), so the registry's identity rules are exercised with genuinely distinct Rust types from one implementation. Two codecs make the paths distinguishable: RefusingCodec errors on every method, so any codec involvement is fatal, and TagCodec can decode the same payload, so a test that should reach it is not merely observing a failure.

• The registry path end to end, including ctx.session().config_options() reading the session's batch size; that value is what tells the two decode paths apart everywhere else in the file.
• A whole plan through physical_plan_to_bytes_with_extension_codec / physical_plan_from_bytes_with_extension_codec.
• Deduplication under DeduplicatingProtoConverter: one expression referenced twice resolves to one deduplicated expression. Arc::ptr_eq on the two decoded expressions is the wrong assertion because the deserializer returns cached.with_new_children(..), a fresh Arc; the test ptr-compares shared state that a fresh decode mints and with_new_children carries over. Dropping the driver's expr_id stamp (the first commit) fails this test on exactly that assertion.
• Nested extension expressions, proving the registry is re-entered for children.
• The fallback rules: an unnamed node reaches the codec and, if the codec fails, reports the codec's own error untouched; a named-but-unregistered name falls back to the codec.
• A registered decoder that errors does not retry through a codec that could decode the node.
• Registry identity: the same type twice is a no-op, a different type under the same name is an error, an empty name is rejected.
• A named node the codec also rejects produces an error naming the expression, how to register it, what is registered, and the codec's own failure as the cause.

Verified locally on the final tree and with cargo check on every commit of the stack: the datafusion-physical-expr, datafusion-physical-plan and datafusion-proto test suites, the examples crate, CI's clippy script, rustdoc with private items under -D warnings, prettier, typos and fmt.

Are there any user-facing changes?

Additive for anyone who does not opt in, with three mechanical breaks documented in the 56.0.0 upgrade guide (one per refactor commit, so they can be reviewed and released independently of the registry): try_from_proto on the built-in expressions is now a trait item, so callers such as BinaryExpr::try_from_proto(..) must import PhysicalExprFromProto; PhysicalExprDecodeCtx gained a session type parameter and new takes the session as a third argument; and the built-in decoders' signatures name PhysicalExprDecodeCtx<'_, ExprDecodeSession<'_>>.

Two things worth a reviewer's attention:

• The wire field is additive for binary protobuf but not for the optional JSON serde: generated JSON deserializers reject unknown fields, so a node carrying exprName needs a reader built after this change.
• The codec fallback is only equivalent for nodes written the old way. Once an expression's try_to_proto writes its own PhysicalExtensionExprNode, a reader that fails to register the type hands that payload to try_decode_expr, and a codec that still recognizes the expression may decode the new payload as the old message rather than fail. The upgrade guide says to register migrated expressions everywhere their plans are read.

🤖 Generated with Claude Code

[github-actions]

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion-physical-expr v55.1.0 (current)
       Built [  40.492s] (current)
     Parsing datafusion-physical-expr v55.1.0 (current)
      Parsed [   0.050s] (current)
    Building datafusion-physical-expr v55.1.0 (baseline)
       Built [  33.976s] (baseline)
     Parsing datafusion-physical-expr v55.1.0 (baseline)
      Parsed [   0.054s] (baseline)
    Checking datafusion-physical-expr v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.345s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure method_requires_different_generic_type_params: method now requires a different number of generic type parameters ---

Description:
A method now requires a different number of generic type parameters than it used to. Uses of this method that supplied the previous number of generic types will be broken.
        ref: https://doc.rust-lang.org/reference/items/generics.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/method_requires_different_generic_type_params.ron

Failed in:
  datafusion_physical_expr::scalar_subquery::ScalarSubqueryExpr::try_from_proto takes 1 generic types instead of 0, in /home/runner/work/datafusion/datafusion/datafusion/physical-expr/src/scalar_subquery.rs:206
  datafusion_physical_expr::Partitioning::try_from_proto takes 1 generic types instead of 0, in /home/runner/work/datafusion/datafusion/datafusion/physical-expr/src/partitioning.rs:824

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  75.980s] datafusion-physical-expr
    Building datafusion-physical-expr-common v55.1.0 (current)
       Built [  27.655s] (current)
     Parsing datafusion-physical-expr-common v55.1.0 (current)
      Parsed [   0.023s] (current)
    Building datafusion-physical-expr-common v55.1.0 (baseline)
       Built [  27.757s] (baseline)
     Parsing datafusion-physical-expr-common v55.1.0 (baseline)
      Parsed [   0.024s] (baseline)
    Checking datafusion-physical-expr-common v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.223s] 223 checks: 217 pass, 6 fail, 0 warn, 31 skip

--- failure function_requires_different_generic_type_params: function now requires a different number of generic type parameters ---

Description:
A function now requires a different number of generic type parameters than it used to. Uses of this function that supplied the previous number of generic types (e.g. via turbofish syntax) will be broken.
        ref: https://doc.rust-lang.org/reference/items/generics.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/function_requires_different_generic_type_params.ron

Failed in:
  function sort_exprs_try_from_proto (0 -> 1 generic types) in /home/runner/work/datafusion/datafusion/datafusion/physical-expr-common/src/sort_expr.rs:276
  function optional_ordering_try_from_proto (0 -> 1 generic types) in /home/runner/work/datafusion/datafusion/datafusion/physical-expr-common/src/sort_expr.rs:298

--- failure method_parameter_count_changed: pub method parameter count changed ---

Description:
A publicly-visible method now takes a different number of parameters, not counting the receiver (self) parameter.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#fn-change-arity
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/method_parameter_count_changed.ron

Failed in:
  datafusion_physical_expr_common::physical_expr::proto_decode::PhysicalExprDecodeCtx::new takes 2 parameters in /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/da347d550619edfbf1f1f395b86615d66cf9f7f7/datafusion/physical-expr-common/src/physical_expr.rs:697, but now takes 3 parameters in /home/runner/work/datafusion/datafusion/datafusion/physical-expr-common/src/physical_expr.rs:714

--- failure method_requires_different_generic_type_params: method now requires a different number of generic type parameters ---

Description:
A method now requires a different number of generic type parameters than it used to. Uses of this method that supplied the previous number of generic types will be broken.
        ref: https://doc.rust-lang.org/reference/items/generics.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/method_requires_different_generic_type_params.ron

Failed in:
  datafusion_physical_expr_common::sort_expr::PhysicalSortExpr::try_from_proto takes 1 generic types instead of 0, in /home/runner/work/datafusion/datafusion/datafusion/physical-expr-common/src/sort_expr.rs:211

--- failure trait_now_doc_hidden: pub trait is now #[doc(hidden)] ---

Description:
A pub trait is now #[doc(hidden)], removing it from the crate's public API.
        ref: https://doc.rust-lang.org/rustdoc/write-documentation/the-doc-attribute.html#hidden
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/trait_now_doc_hidden.ron

Failed in:
  trait PhysicalExprDecode in file /home/runner/work/datafusion/datafusion/datafusion/physical-expr-common/src/physical_expr.rs:826
  trait PhysicalExprEncode in file /home/runner/work/datafusion/datafusion/datafusion/physical-expr-common/src/physical_expr.rs:608

--- failure trait_requires_more_generic_type_params: trait now requires more generic type parameters ---

Description:
A trait now requires more generic type parameters than it used to. Uses of this trait that supplied the previously-required number of generic types will be broken. To fix this, consider supplying default values for newly-added generic types.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#trait-new-parameter-no-default
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/trait_requires_more_generic_type_params.ron

Failed in:
  trait PhysicalExprDecodeCtx (0 -> 1 required generic types) in /home/runner/work/datafusion/datafusion/datafusion/physical-expr-common/src/physical_expr.rs:694

--- failure type_requires_more_generic_type_params: type now requires more generic type parameters ---

Description:
A type now requires more generic type parameters than it used to. Uses of this type that supplied the previously-required number of generic types will be broken. To fix this, consider supplying default values for newly-added generic types.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#trait-new-parameter-no-default
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/type_requires_more_generic_type_params.ron

Failed in:
  Struct PhysicalExprDecodeCtx (0 -> 1 required generic types) in /home/runner/work/datafusion/datafusion/datafusion/physical-expr-common/src/physical_expr.rs:694

     Summary semver requires new major version: 6 major and 0 minor checks failed
    Finished [  56.477s] datafusion-physical-expr-common
    Building datafusion-physical-plan v55.1.0 (current)
       Built [  43.248s] (current)
     Parsing datafusion-physical-plan v55.1.0 (current)
      Parsed [   0.177s] (current)
    Building datafusion-physical-plan v55.1.0 (baseline)
       Built [  44.547s] (baseline)
     Parsing datafusion-physical-plan v55.1.0 (baseline)
      Parsed [   0.182s] (baseline)
    Checking datafusion-physical-plan v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.692s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  90.368s] datafusion-physical-plan
    Building datafusion-proto v55.1.0 (current)
       Built [  64.361s] (current)
     Parsing datafusion-proto v55.1.0 (current)
      Parsed [   0.025s] (current)
    Building datafusion-proto v55.1.0 (baseline)
       Built [  64.275s] (baseline)
     Parsing datafusion-proto v55.1.0 (baseline)
      Parsed [   0.019s] (baseline)
    Checking datafusion-proto v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.121s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 130.094s] datafusion-proto
    Building datafusion-proto-models v55.1.0 (current)
       Built [  28.926s] (current)
     Parsing datafusion-proto-models v55.1.0 (current)
      Parsed [   0.138s] (current)
    Building datafusion-proto-models v55.1.0 (baseline)
       Built [  28.672s] (baseline)
     Parsing datafusion-proto-models v55.1.0 (baseline)
      Parsed [   0.142s] (baseline)
    Checking datafusion-proto-models v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   1.852s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure constructible_struct_adds_field: struct exhaustively constructible through public API adds field ---

Description:
A pub struct that could be exhaustively constructed with a literal using only public API has a new pub field, breaking existing exhaustive literals.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field PhysicalExtensionExprNode.expr_name in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1934
  field PhysicalExtensionExprNode.expr_name in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1934
  field PhysicalExtensionNode.plan_name in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1610
  field PhysicalExtensionNode.plan_name in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1610

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  60.850s] datafusion-proto-models

[adriangb]

Should this be a method on PhysicalExpr like to_proto?

[codecov-commenter]

Codecov Report

❌ Patch coverage is 86.64073% with 103 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.46%. Comparing base (7570366) to head (f11996d).
⚠️ Report is 19 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/physical-plan/src/proto_test_util.rs	51.38%	34 Missing and 1 partial ⚠️
datafusion/proto-models/src/generated/pbjson.rs	0.00%	26 Missing ⚠️
datafusion/proto-models/src/registry.rs	87.50%	13 Missing and 5 partials ⚠️
datafusion/physical-expr/src/proto.rs	81.25%	7 Missing and 2 partials ⚠️
datafusion/physical-plan/src/proto/registry.rs	87.67%	1 Missing and 8 partials ⚠️
datafusion/proto/src/physical_plan/mod.rs	96.00%	2 Missing ⚠️
...atafusion/physical-expr/src/expressions/in_list.rs	96.55%	1 Missing ⚠️
datafusion/physical-plan/src/proto/mod.rs	98.55%	1 Missing ⚠️
datafusion/proto/src/physical_plan/from_proto.rs	96.96%	1 Missing ⚠️
datafusion/proto/src/physical_plan/to_proto.rs	90.90%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##             main   #24628    +/-   ##
========================================
  Coverage   82.45%   82.46%            
========================================
  Files        1140     1143     +3     
  Lines      436589   437187   +598     
  Branches   436589   437187   +598     
========================================
+ Hits       359996   360514   +518     
- Misses      54838    54900    +62     
- Partials    21755    21773    +18     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[adriangb]

I put this in draft. Lets sequence behind #24631 to avoid duplication. I will keep it updated though.