Skip to content

Expose hidden JVM function execution inside native Calc - #314

Merged
liuyongvs merged 1 commit into
datafusion-contrib:mainfrom
ChaomingZhangCN:codex/native-jvm-expression-logging
Oct 10, 2026
Merged

liuyongvs merged 1 commit into
datafusion-contrib:mainfrom
ChaomingZhangCN:codex/native-jvm-expression-logging

Conversation

@ChaomingZhangCN

Copy link
Copy Markdown

Why

A NativeCalc or NativeFilter in the physical plan does not necessarily mean that every scalar expression runs in Rust. The node can still contain host-exact scalar upcalls or a whole-Calc Flink-generated evaluator.

Previously, the planner summary only reported operator substitution. A plan could therefore report that all operators were substituted natively while some functions still executed on the JVM, with no compile-time diagnostic explaining that placement.

What changed

  • Record the final execution engine assignment for functions in admitted Calc and filter nodes.
  • Log distinct function calls under native and jvm.
  • Preserve the reason when host-exact evaluation or whole-Calc JVM evaluation is selected.
  • Count Calc/filter nodes containing JVM functions in the always-on planner summary.
  • Roll back provisional diagnostics when the surrounding native island is rejected.
  • Show SQL functions, UDFs, and casts, while omitting arithmetic, comparison, and boolean operators.

With -Dstreamfusion.logFallbackReasons=true, function assignments are emitted at INFO; otherwise they remain available at DEBUG.

Example mixed execution:

NativeCalc functions: native=[CAST, CHAR_LENGTH], jvm=[JVM_ONLY],
reason=host-exact or scalar-UDF implementation selected

Example whole-Calc JVM execution:

NativeCalc functions: native=[], jvm=[IF],
reason=multiple fallible/JVM expressions require Flink's row-major error order

The planner summary now also reports the number of admitted Calc/filter nodes that contain JVM functions.

Testing

  • Added coverage for fully native, mixed native/JVM, and whole-Calc JVM execution.
  • Added coverage for multiple IF expressions requiring row-major JVM evaluation.
  • Verified that casts remain visible while arithmetic and comparison operators are omitted.
  • Targeted planner and generated-expression suite: 20 tests passed on Flink 2.2.1 / Java 17.

A native plan node can still contain host-exact scalar evaluation, so operator substitution alone is not enough to explain execution placement. Record the final function-to-engine assignment after planner admission, retain the reason for whole-Calc JVM scheduling, and summarize nodes with JVM work at compile time.\n\nKeep the diagnostic focused on SQL functions, UDFs, and casts whose semantics can change placement; omit arithmetic, comparison, and boolean operators.
@ChaomingZhangCN

Copy link
Copy Markdown
Author

@liuyongvs PTAL~

@liuyongvs liuyongvs left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM +1

@liuyongvs
liuyongvs merged commit 9724b9d into datafusion-contrib:main Oct 10, 2026
36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants