Repository navigation
Add vector distance, array math, and array aggregate functions #21536
Description
Summary
This issue tracks adding vector math and array aggregate scalar functions to DataFusion. These close gaps versus DuckDB and LanceDB for vector search and array analytics workloads.
Replaces #21371 and #21376, which were requested to be split into function-per-PR submissions (per @alamb's review).
Functions
Vector math (with shared vector_math.rs primitives)
| Function | Signature | Reference |
|---|---|---|
cosine_distance |
(array, array) → float64 |
DuckDB array_cosine_similarity |
inner_product |
(array, array) → float64 |
DuckDB array_inner_product |
array_normalize |
(array) → array |
NumPy / scipy convention |
Array element-wise math
| Function | Signature | Reference |
|---|---|---|
array_add |
(array, array) → array |
Element-wise addition |
array_subtract |
(array, array) → array |
Element-wise subtraction |
array_scale |
(array, scalar) → array |
Scalar multiply |
Array aggregate scalars
| Function | Signature | Reference |
|---|---|---|
array_sum / list_sum |
(array) → numeric |
DuckDB list_sum |
array_product / list_product |
(array) → numeric |
DuckDB list_product |
array_avg / list_avg |
(array) → float64 |
DuckDB list_avg |
Alias fix
| Fix | Description |
|---|---|
list_min |
Missing alias on ArrayMin (parity with existing list_max on ArrayMax) |
Submission plan
One PR per function, submitted serially. Each PR will reference this issue.
Activity
feat: add cosine_distance scalar function
feat: rework cosine_distance per review feedback
feat: handle mixed-type and NULL inputs for cosine_distance
feat: add cosine_distance scalar function (apache#21542)
alamb commentedon Apr 27, 2026
Given the DataFusion core is pretty large already, i wonder if we would be better off making this package of functions as a datafusion-contrib repo
Perhaps something like https://github.com/datafusion-contrib/datafusion-functions-json 🤔
At least until we have a larger community of poeple / committers who are willing to help wiht maintaining a new function library
I realize this is a bit late as we just merged #21542
But I am thinking that
datafusion-contrib/array_mathmight make a very nice function package 🤔Jefffrey commentedon Apr 28, 2026
I feel the functions suggested here are simple enough that maintenance burden isn't too much a concern
feat: add inner_product scalar function (apache#21861)
feat: add array_normalize scalar function (apache#22013)
SubhamSinghal commentedon May 22, 2026
@crm26 I would like to help on this.
crm26 commentedon May 22, 2026
Hey @SubhamSinghal — thanks for reaching out and happy to collaborate however works.
Six functions left:
array_add,array_subtract,array_scale,array_sum,array_product,array_avg. The three already merged (#21542, #21861, #22013) lock in the per-function pattern Jefffrey's been reviewing.Heads up: I've been going serially (one PR open at a time) since alamb's original split request, so we don't pile work on reviewers. Three of my older PRs (#21363, #21353, #21367) are also still stale. Worth keeping the in-flight count modest between us — maybe 2 open at a time across both contributors?
How would you like to approach it?
SubhamSinghal commentedon May 22, 2026
@crm26 sound fine to me. You can specify which one you are working on. I can work on rest.
feat: add array_add scalar function
feat: add array_add scalar function
feat: add array_add scalar function
crm26 commentedon May 22, 2026
Hey @SubhamSinghal — saw #22459, thanks for the fast turnaround.
array_subtractis essentially a copy ofarray_add(just operator swap), so happy for you to take that next too. I'll claimarray_scale(different signature — list + scalar — so it covers new territory and we don't collide). The three aggregates (array_sum,array_product,array_avg) are still open for whoever has bandwidth.Will leave any comments on #22459 directly if I spot anything as I read through.
7 remaining items
feat: add array_scale scalar function
feat: add array_scale scalar function
feat: add array_scale scalar function (apache#22466)
feat: add array_sum scalar function
feat: add array_sum scalar function
feat: add array_sum scalar function
feat: add array_sum scalar function
feat: adds array_add function (apache#22459)
feat: add array_subtract scalar function (apache#22556)
Add `array_product` UDF (apache#22703)
feat: add array_sum scalar function
feat: add array_sum scalar function (apache#22542)
feat: add array_sum scalar function (apache#22542)
crm26 commentedon Jun 24, 2026
Claiming
array_avg— the last unclaimed split function from this issue. Will template from the mergedarray_sum(#22542) using the same SQL aggregate NULL convention: skip NULL elements, empty array → NULL, all-NULL row → NULL. Result typeFloat64. Aliaslist_avgper thelist_sum/list_productpattern.feat: add array_avg scalar function (apache#23168)