feat: Use NativeType in get_example_type, information schema by theirix · Pull Request #21737 · apache/datafusion

theirix · 2026-04-19T20:04:02Z

Which issue does this PR close?

Closes Return NativeType instead of DataType for get_example_types #14761

Rationale for this change

Moving from physical types: get_example_types and the information schema use Arrow DataType, but it is usually sufficient to use Datafusion's NativeType instead.

Let's introduce a new API get_representative_types, based on NativeType, and deprecate the public get_example_types API

It is a logical continuation of #15965

What changes are included in this PR?

Introduce get_representative_types to provide NativeType via LogicalType::default_cast_for
Add missing conversions for Union to that helper
Deprecate NUMERICS finally (a brush-up for Refactor away usage of NUMERICS/INTEGERS in datafusion/expr-common/src/type_coercion/aggregates.rs #18092)
Switch information schema to use native type.

Are these changes tested?

Tests are passing
Added more tests to check schema for different types of functions

Are there any user-facing changes?

Deprecation of the TypeSignature::get_example_types API

martin-g · 2026-04-20T06:53:28Z


    #[deprecated(since = "46.0.0", note = "See get_example_types instead")]
-    pub fn get_possible_types(&self) -> Vec<Vec<DataType>> {
+    pub fn get_possible_types(&self) -> Vec<Vec<NativeType>> {


Hm. A public API break for a deprecated method ... Maybe it is time to just remove it ?!

That's true. I will drop get_possible_types. However, get_example_types is also public - it is used in the information schema internally, and can be used by DF users.

Since it breaks the expr-common API, how about deprecating the get_example_types signature and adding a NativeType-based get_representative_types?

@martin-g to avoid breaking changes in the get_example_types (my miss), I've introduced the new method, while deprecating get_example_types.

The Signature API is heavily DataType-based, so the get_example_types. Should we let them coexist, or even move the NativeType-based logic to the information schema? cc @jayzhan211

It seems the NativeType is not powerful enough to fully replace DataType here.
For example https://github.com/apache/datafusion/pull/21737/changes#diff-52d29120f24c2a01793f6d729fdb7898abaa8d2320db75aba5cbb06cd714930eR505-R516 shows that there are no counterparts for Union's UnionMode and Map's keys_sorted and defaults should be used. And this may lead to confusions.

Undoubtedly, it's not a replacement now. The question for this improvement is whether we'd like to provide changes only to the information-schema, which benefits from native types (then we move code there), or also provide a public API for the Signature to support NativeType alongside DataType (as now).

At the same time, UDFs are migrating to TypeSignature, abstracting from physical data types.

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

Instead, add the new `get_representative_types` API with NativeType. Deprecated old `get_example_types` and related helpers, left as-is to avoid breaking API change.

martin-g · 2026-04-22T13:36:13Z

    ///
    /// This is used for `information_schema` and can be used to generate
    /// documentation or error messages.
+    /// Remove with `get_example_types`


This should be a comment (//) instead of rustdoc (///)

martin-g · 2026-04-22T13:43:52Z


    #[deprecated(since = "46.0.0", note = "See get_example_types instead")]
-    pub fn get_possible_types(&self) -> Vec<Vec<DataType>> {
+    pub fn get_possible_types(&self) -> Vec<Vec<NativeType>> {


It seems the NativeType is not powerful enough to fully replace DataType here.
For example https://github.com/apache/datafusion/pull/21737/changes#diff-52d29120f24c2a01793f6d729fdb7898abaa8d2320db75aba5cbb06cd714930eR505-R516 shows that there are no counterparts for Union's UnionMode and Map's keys_sorted and defaults should be used. And this may lead to confusions.

Jefffrey

sorry it took so long for me to get around to looking at this 😅

Jefffrey · 2026-06-14T03:44:21Z

+        NativeType::Boolean => Ok(DataType::Boolean),
+        NativeType::Int8 => Ok(DataType::Int8),
+        NativeType::Int16 => Ok(DataType::Int16),
+        NativeType::Int32 => Ok(DataType::Int32),


It feels like we're just moving this logic from where it was in get_example_types to here; I think to properly work towards closing the original issue we might need a larger rework around how datatypes work with information schema, otherwise we're still stuck with this datatype <---> nativetype interwork no matter where we move it 🤔

@Jefffrey , thank you for this! I took another look and reworked it significantly.

I agree, UDF's field and return type resolvers are still on physical Arrow types, and the high-level rework should be done as part of the bigger #12622 epic. I think this could be a small step toward that migration.

Instead of moving a huge mapping around, I decided to reuse the existing LogicalType::default_cast_for logical-physical mapping. It is much leaner now, and all the logic is consolidated in one place, not spread across crates. We can reason on logical types for the catalog. When we remove the deprecated get_example_types, there won't be any traces of the mapping in the high-level catalog and expr crates. What do you think?

@martin-g, regarding your concern - the details of mapping, keys_sorted, and nested structures are now the responsibility of a LogicalType

github-actions · 2026-08-14T02:14:09Z

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.

github-actions · 2026-08-18T13:10:53Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion-catalog v55.1.0 (current)
       Built [  27.633s] (current)
     Parsing datafusion-catalog v55.1.0 (current)
      Parsed [   0.016s] (current)
    Building datafusion-catalog v55.1.0 (baseline)
       Built [  25.465s] (baseline)
     Parsing datafusion-catalog v55.1.0 (baseline)
      Parsed [   0.016s] (baseline)
    Checking datafusion-catalog v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.083s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  54.712s] datafusion-catalog
    Building datafusion-common v55.1.0 (current)
       Built [  21.623s] (current)
     Parsing datafusion-common v55.1.0 (current)
      Parsed [   0.045s] (current)
    Building datafusion-common v55.1.0 (baseline)
       Built [  21.151s] (baseline)
     Parsing datafusion-common v55.1.0 (baseline)
      Parsed [   0.043s] (baseline)
    Checking datafusion-common v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.528s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  44.497s] datafusion-common
    Building datafusion-expr-common v55.1.0 (current)
       Built [  11.711s] (current)
     Parsing datafusion-expr-common v55.1.0 (current)
      Parsed [   0.012s] (current)
    Building datafusion-expr-common v55.1.0 (baseline)
       Built [  11.617s] (baseline)
     Parsing datafusion-expr-common v55.1.0 (baseline)
      Parsed [   0.012s] (baseline)
    Checking datafusion-expr-common v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.157s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure type_method_marked_deprecated: type method #[deprecated] added ---

Description:
A type method is now #[deprecated]. Downstream crates will get a compiler warning when using this method.
        ref: https://doc.rust-lang.org/reference/attributes/diagnostics.html#the-deprecated-attribute
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/type_method_marked_deprecated.ron

Failed in:
  method datafusion_expr_common::signature::TypeSignature::get_example_types in /home/runner/work/datafusion/datafusion/datafusion/expr-common/src/signature.rs:937

     Summary semver requires new minor version: 0 major and 1 minor checks failed
    Finished [  24.245s] datafusion-expr-common
    Building datafusion-sqllogictest v55.1.0 (current)
       Built [  56.702s] (current)
     Parsing datafusion-sqllogictest v55.1.0 (current)
      Parsed [   0.011s] (current)
    Building datafusion-sqllogictest v55.1.0 (baseline)
       Built [  57.278s] (baseline)
     Parsing datafusion-sqllogictest v55.1.0 (baseline)
      Parsed [   0.012s] (baseline)
    Checking datafusion-sqllogictest v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.066s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 116.666s] datafusion-sqllogictest

codecov-commenter · 2026-08-18T13:27:53Z

Codecov Report

❌ Patch coverage is 97.44409% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.66%. Comparing base (6a792c6) to head (f8a88d9).
⚠️ Report is 107 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/catalog/src/information_schema.rs	93.93%	3 Missing and 3 partials ⚠️
datafusion/common/src/types/native.rs	97.64%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #21737      +/-   ##
==========================================
+ Coverage   82.49%   82.66%   +0.16%     
==========================================
  Files        1140     1147       +7     
  Lines      438555   446575    +8020     
  Branches   438555   446575    +8020     
==========================================
+ Hits       361803   369176    +7373     
- Misses      54880    54997     +117     
- Partials    21872    22402     +530

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

theirix · 2026-08-18T15:06:59Z

bot unstale ?

Jefffrey · 2026-09-24T10:14:09Z

maybe @jayzhan211 can help take a look at this

jayzhan211

Thanks @theirix , here is a suggestion

jayzhan211 · 2026-09-24T12:18:32Z

+                    .collect::<Result<Vec<FieldRef>>>()?;
+                // Even with collecting results into a set, drop duplicates early
+                arg_fields.sort();
+                arg_fields.dedup();


Field's Ord compares name first, and every field is named arg_{i}, so dedup() never removes anything. sort() is a no-op up to 10 args, but from 11 args it reorders them lexicographically (arg_0, arg_1, arg_10, arg_2, …). The return type is then computed from permuted argument types, while the arg_types shown keep the original order.

Repro: a UDF with Signature::exact(vec![DataType::Int32; 10] + [DataType::Utf8]) whose return_type returns the last arg. get_udf_args_and_return_types reports args [Int32 ×10, String] with return type Some("Int32") instead of Some("String").

The BTreeSet collect already dedups rows, so drop both lines:

- let mut arg_fields = arg_types + let arg_fields = arg_types .iter() .enumerate() .map(|(i, t)| resolve_informational_field(i, t)) .collect::<Result<Vec<FieldRef>>>()?; - // Even with collecting results into a set, drop duplicates early - arg_fields.sort(); - arg_fields.dedup();

Please add a test with more than 10 heterogeneous args.

Thank you, it's a good spot. Removed the dedup to have an expected result (backed by a unit test)

jayzhan211

Thanks @theirix , here are suggestions

jayzhan211 · 2026-10-01T13:04:46Z

+fn resolve_informational_field(idx: usize, t: &NativeType) -> Result<FieldRef> {
+    // Since a native type maps to several physical types, resolve it against `Null` data type
+    // to get the canonical `DataType` for the native type
+    let data_type = t.default_cast_for(&DataType::Null)?;


Resolving each NativeType against Null collapses String to Utf8View, so string functions that return Int64 for LargeUtf8 lose that row. Diffing information_schema.routines/parameters against the base shows the Int64 OUT row disappearing for bit_length, char_length, character_length, find_in_set, instr, length, levenshtein, octet_length, position, strpos. If that's intended, please pin it in information_schema.slt and mention it under user-facing changes:

query TT rowsort select routine_name, data_type from information_schema.routines where routine_name = 'length'; ---- length Int32

jayzhan211 · 2026-10-01T13:05:20Z

Do we need this change?

Use NativeType in get_example_types and information schema

2d47696

github-actions Bot added logical-expr Logical plan and expressions catalog Related to the catalog crate labels Apr 19, 2026

theirix marked this pull request as ready for review April 19, 2026 20:28

martin-g reviewed Apr 20, 2026

View reviewed changes

theirix and others added 5 commits April 20, 2026 20:08

Apply suggestions from code review

179f794

Co-authored-by: Martin Grigorov <martin-g@users.noreply.github.com>

Use of Decimals

a22c2f6

Refactor get_data_type_for_schema to return result

c54da40

Remove old deprecated get_possible_types

50b6858

Avoid breaking get_example_types API

5276933

Instead, add the new `get_representative_types` API with NativeType. Deprecated old `get_example_types` and related helpers, left as-is to avoid breaking API change.

martin-g reviewed Apr 22, 2026

View reviewed changes

Fix docs

840fbb9

Jefffrey reviewed Jun 14, 2026

View reviewed changes

github-actions Bot added the Stale PR has not had any activity for some time label Aug 14, 2026

theirix added 2 commits August 18, 2026 09:31

Merge branch 'main' into type-signature-native-type

42ee8ee

Reuse LogicalType type mapping

687fa6a

github-actions Bot added sqllogictest SQL Logic Tests (.slt) common Related to common crate labels Aug 18, 2026

github-actions Bot added the auto detected api change Auto detected API change label Aug 18, 2026

Format

3d1b43a

github-actions Bot removed the Stale PR has not had any activity for some time label Aug 19, 2026

jayzhan211 reviewed Sep 24, 2026

View reviewed changes

theirix added 3 commits September 26, 2026 11:20

Merge branch 'main' into type-signature-native-type

123bec2

Sync test cases with main

6997029

Avoid dedup with arg sorting

7fff35a

jayzhan211 reviewed Oct 1, 2026

View reviewed changes

theirix added 4 commits October 4, 2026 22:14

Resolve ambigious native-to-physical types

736e56d

Test ambigious cases

082df11

Avoid committing parquet-testing

0c8e779

Typos

f8a88d9

Conversation

theirix commented Apr 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jefffrey left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 14, 2026

Uh oh!

github-actions Bot commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

theirix commented Aug 18, 2026

Uh oh!

Jefffrey commented Sep 24, 2026

Uh oh!

jayzhan211 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jayzhan211 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

theirix commented Apr 19, 2026 •

edited

Loading

github-actions Bot commented Aug 18, 2026 •

edited

Loading

codecov-commenter commented Aug 18, 2026 •

edited

Loading