Skip to content

[C++] Add a "list_contains" kernel #33295

Open
#51611
Open
[C++] Add a "list_contains" kernel#33295
#51611
Assignees
jonasdedden
Labels
Component: C++Status: needs championHigh impact issues which aren't being worked on but require a volunteer to move the task forward.Type: enhancement

Description

@asfimport
asfimport
opened on Oct 19, 2022

Assume you have a list array:

arr = pa.array([["a", "b"], ["a", "c"], ["b", "c", "d"]])

And you want to know for each list if it contains a certain value (of the same type as the list's values). A "list_contains" function (or other name) would be useful for that:

pc.list_contains(arr, "a")
# -> True, True False

The current workaround that I found was flattening, checking equality, and then reducing again with groupby, but this is quite tedious:

>>> temp = pa.table({'index': pc.list_parent_indices(arr), 'contains_value': pc.equal(pc.list_flatten(arr), "a")})
>>> temp.group_by('index').aggregate([('contains_value', 'any')])['contains_value_any'].chunk(0)
<pyarrow.lib.BooleanArray object at 0x7ffaf3f8de20>
[
  true,
  true,
  false
]

But this also only works if there are no empty or missing list values.

Reporter: Joris Van den Bossche / @jorisvandenbossche

Related issues:

Note: This issue was originally created as ARROW-18097. Please see the migration documentation for further details.

Activity

asfimport commented on Oct 19, 2022

@asfimport
asfimport
on Oct 19, 2022
Author

Antoine Pitrou / @pitrou:
Then there probably should be a "list_index" function as well, similar to "is_in" vs. "index_in" ?

pc.list_index(arr, "b")
# -> 1, None, 0

asfimport commented on Nov 2, 2022

@asfimport
asfimport
on Nov 2, 2022
Author

Jacek Pliszka / @JacekPliszka:
Maybe no need for new name but is_in can be reused:

pc.is_in("a", arr)

asfimport commented on Nov 3, 2022

@asfimport
asfimport
on Nov 3, 2022
Author

Joris Van den Bossche / @jorisvandenbossche:
[~jacek.pliszka] That would give a different signature; "is_in" is currently a scalar kernel that returns a result of the same shape as the first input argument. So that generally wouldn't work here (for lists, it would the second argument that we want to check element-by-element in your example).
While it might be technically possible in C++ to have kernels with different input types, I am not sure it is a good idea to reuse the same name for this.

thisisnic
added
Status: stale-warningIssues and PRs flagged as stale which are due to be closed if no indication otherwise
on Feb 18, 2026

thisisnic commented on Feb 18, 2026

@thisisnic
thisisnic
on Feb 18, 2026
Member

This issue has been marked as stale because it has had no activity in the past 2 years. Please remove the stale label or comment below, or this issue will be closed in 14 days.

alexshpilkin commented on Feb 18, 2026

@alexshpilkin
alexshpilkin
on Feb 18, 2026

Mere passage of time does not cause a feature to appear. Not stale.

thisisnic commented on Feb 18, 2026

@thisisnic
thisisnic
on Feb 18, 2026
Member

Thanks @alexshpilkin, will mark as not stale. Are you interested in submitting a PR?

thisisnic
added
Status: needs championHigh impact issues which aren't being worked on but require a volunteer to move the task forward.
and removed
Status: stale-warningIssues and PRs flagged as stale which are due to be closed if no indication otherwise
on Feb 18, 2026

alexshpilkin commented on Feb 18, 2026

@alexshpilkin
alexshpilkin
on Feb 18, 2026

@thisisnic This was one of the issue that pushed me to Polars (not the issue, by any means, just one of the things that contributed to the decision). So it’s less than immediately relevant to me at the moment. Still, I’m not against giving it a shot in principle, if you or someone else could point me in the right direction. I just don’t promise the kernel will be particularly good :)

thisisnic commented on Feb 18, 2026

@thisisnic
thisisnic
on Feb 18, 2026
Member

If you make a start and draft a PR, folks around here are pretty friendly and happy to help, just pop some comments in explaining where you got to and where you aren't sure :) Perfection is not a requirement, for sure!

thisisnic
assigned
alexshpilkin
on Feb 18, 2026

jonasdedden commented on Sep 28, 2026

@jonasdedden
jonasdedden
on Sep 28, 2026
Contributor

@pitrou @jorisvandenbossche @thisisnic please have a look at #51611, where I implement a list.contains compute kernel.

jonasdedden
added 7 commits that reference this issue on Sep 28, 2026

apacheGH-33295: [C++] Support arrays of values in list_contains

7349c0e

apacheGH-33295: [C++] Type-check null values in list_contains

7004d33

apacheGH-33295: [C++] Match float16 NaN values in list_contains

cb3cea8

apacheGH-33295: [C++] Only compare child values referenced by list vi…

441fdb3

apacheGH-33295: [C++] Stop at the first match in list_contains

acc42f1

apacheGH-33295: [C++] Share one code path for all list types and valu…

2d1b613

apacheGH-33295: [C++] Condense list_contains tests

95d6b87
tadeja
assigned
jonasdedden
and unassigned
alexshpilkin
on Sep 29, 2026
jonasdedden
added a commit that references this issue on Sep 30, 2026

apacheGH-33295: [C++] Speed up the list_contains scan for short lists

972bc4c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Component: C++Status: needs championHigh impact issues which aren't being worked on but require a volunteer to move the task forward.Type: enhancement

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions