Repository navigation
Commit 4e90755
authored
## Which issue does this PR close?
- Closes #25083.
- Closes #25210.
## Rationale for this change
`map[key]` (aka `get_field`) and `map_extract(map, key)` collectively
had three ways to lookup keys in a map:
(1): `get_field` used a per-batch `eq` kernel for scalar keys. This is
efficient for maps with many entries where `key` is not found, but
slower for maps in which the key can be found quickly (because `eq`
does not allow early-stopping). On my local machine, `eq` only
beats a comparator-based approach if the latter required touching
more than 60% of the keys in a row.
(2): `get_field` used a comparator-based approach for nested keys.
(3): `map_extract` always used a comparator-based approach.
Conceptually, these two functions only differ in how the result is
represented, so it makes sense to consolidate them. We can also adopt a
hybrid strategy that gets the best of the previous approaches for most
inputs:
* Start with a comparator-based approach.
* After the first row, remember the index at which the matching key was
found, and check that index first for subsequent rows. This takes
advantage of the observation that most map rows have their keys in
the same order.
* After 32 rows, check whether the comparator looked at more than 50% of
the entries in those rows. If it did, the early stopping that the
comparator approach allows is not useful and we switch to an `eq`
kernel for the remainder of the batch (as long as the map key is not a
nested type).
We can also use `take` to construct the results, which is faster than
the previous approach based on `MutableArrayData::extend`.
Along the way, fix `get_field` to behave correctly for NULL map rows
with non-empty offset ranges (#25210).
Benchmarks: (M4 Max)
get_field (map[key]):
- get_field_map/int32/last/1024x4: 8.13 µs -> 3.52 µs, -56.7%
- get_field_map/int32/shuffled/1024x4: 7.81 µs -> 4.34 µs, -44.5%
- get_field_map/int32/missing/1024x4: 7.91 µs -> 4.86 µs, -38.5%
- get_field_map/int32/first/1024x32: 9.73 µs -> 3.56 µs, -63.4%
- get_field_map/int32/last/1024x32: 20.19 µs -> 3.64 µs, -82.0%
- get_field_map/int32/shuffled/1024x32: 15.03 µs -> 13.09 µs, -12.9%
- get_field_map/int32/missing/1024x32: 18.70 µs -> 19.91 µs, +6.5%
- get_field_map/utf8_view/last/1024x4: 15.34 µs -> 5.74 µs, -62.6%
- get_field_map/utf8_view/shuffled/1024x4: 15.61 µs -> 12.47 µs, -20.1%
- get_field_map/utf8_view/missing/1024x4: 14.97 µs -> 11.93 µs, -20.3%
- get_field_map/utf8_view/first/1024x32: 68.91 µs -> 6.38 µs, -90.7%
- get_field_map/utf8_view/last/1024x32: 83.72 µs -> 6.63 µs, -92.1%
- get_field_map/utf8_view/shuffled/1024x32: 79.27 µs -> 76.77 µs, -3.2%
- get_field_map/utf8_view/missing/1024x32: 71.72 µs -> 72.85 µs, +1.6%
- get_field_map/struct/last/1024x4: 13.52 µs -> 4.06 µs, -70.0%
- get_field_map/struct/shuffled/1024x4: 10.75 µs -> 9.01 µs, -16.2%
- get_field_map/struct/missing/1024x4: 12.88 µs -> 10.71 µs, -16.9%
- get_field_map/struct/first/1024x32: 8.41 µs -> 4.13 µs, -50.9%
- get_field_map/struct/last/1024x32: 64.71 µs -> 4.24 µs, -93.5%
- get_field_map/struct/shuffled/1024x32: 38.84 µs -> 36.01 µs, -7.3%
- get_field_map/struct/missing/1024x32: 59.14 µs -> 58.04 µs, -1.9%
map_extract:
- map_extract/int32/last/1x0: 323 ns -> 420 ns, +30.0%
- map_extract/int32/last/1x1: 382 ns -> 372 ns, -2.6%
- map_extract/int32/last/1024x4: 9.40 µs -> 3.99 µs, -57.5%
- map_extract/int32/shuffled/1024x4: 8.27 µs -> 4.56 µs, -44.8%
- map_extract/int32/missing/1024x4: 5.54 µs -> 5.58 µs, +0.7%
- map_extract/int32/varying/1024x4: 7.61 µs -> 7.22 µs, -5.1%
- map_extract/utf8_view/last/1024x4: 20.07 µs -> 6.46 µs, -67.8%
- map_extract/utf8_view/shuffled/1024x4: 14.91 µs -> 12.46 µs, -16.4%
- map_extract/utf8_view/missing/1024x4: 16.15 µs -> 13.30 µs, -17.6%
- map_extract/utf8_view/varying/1024x4: 14.69 µs -> 16.21 µs, +10.4%
- map_extract/struct/last/1024x4: 14.07 µs -> 4.49 µs, -68.1%
- map_extract/struct/shuffled/1024x4: 11.58 µs -> 9.43 µs, -18.6%
- map_extract/struct/missing/1024x4: 10.30 µs -> 11.44 µs, +11.0%
- map_extract/struct/varying/1024x4: 9.96 µs -> 10.69 µs, +7.3%
- map_extract/int32/first/1024x32: 6.26 µs -> 4.26 µs, -31.9%
- map_extract/int32/last/1024x32: 38.53 µs -> 4.44 µs, -88.5%
- map_extract/int32/shuffled/1024x32: 25.13 µs -> 13.95 µs, -44.5%
- map_extract/int32/missing/1024x32: 34.91 µs -> 21.73 µs, -37.7%
- map_extract/int32/varying/1024x32: 25.49 µs -> 24.45 µs, -4.1%
- map_extract/utf8_view/first/1024x32: 10.15 µs -> 6.86 µs, -32.4%
- map_extract/utf8_view/last/1024x32: 122.54 µs -> 7.03 µs, -94.3%
- map_extract/utf8_view/shuffled/1024x32: 71.11 µs -> 85.15 µs, +19.8%
- map_extract/utf8_view/missing/1024x32: 113.40 µs -> 81.12 µs, -28.5%
- map_extract/utf8_view/varying/1024x32: 71.74 µs -> 66.87 µs, -6.8%
- map_extract/struct/first/1024x32: 8.62 µs -> 4.79 µs, -44.4%
- map_extract/struct/last/1024x32: 70.64 µs -> 4.97 µs, -93.0%
- map_extract/struct/shuffled/1024x32: 41.94 µs -> 41.52 µs, -1.0%
- map_extract/struct/missing/1024x32: 61.08 µs -> 63.58 µs, +4.1%
- map_extract/struct/varying/1024x32: 40.54 µs -> 38.60 µs, -4.8%
## What changes are included in this PR?
* Add `datafusion_functions::utils::map_lookup`; for each map row, this
returns the index of a matching entry or null.
* Implement `get_field` and `map_extract` on top of the shared
`map_lookup` helper, constructing the results with `take`
* In `map_extract`, optimize for the single-scalar-key case by passing
it through as a scalar value instead of expanding it to the batch size
* `map[NULL]` now returns `NULL` instead of producing an error
* Fix #25210
* Overhaul benchmarks
## What is the testing strategy for this PR?
Existing tests pass; new tests added.
## Are there any user-facing changes?
No, aside from the bugfix and some corner-case changes like how error
messages are formatted.
1 parent 28c0dec commit 4e90755
6 files changed
Lines changed: 869 additions & 270 deletions
File tree
- datafusion
- functions-nested
- benches
- src
- functions
- benches/misc
- src
- core
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
210 | 210 | | |
211 | 211 | | |
212 | 212 | | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
213 | 216 | | |
214 | 217 | | |
215 | 218 | | |
216 | 219 | | |
217 | | - | |
218 | | - | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
219 | 229 | | |
220 | 230 | | |
221 | 231 | | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
222 | 235 | | |
223 | 236 | | |
224 | 237 | | |
| |||
229 | 242 | | |
230 | 243 | | |
231 | 244 | | |
232 | | - | |
233 | | - | |
234 | | - | |
235 | | - | |
236 | | - | |
237 | | - | |
238 | | - | |
239 | | - | |
240 | | - | |
241 | | - | |
242 | | - | |
243 | | - | |
244 | | - | |
245 | | - | |
246 | | - | |
247 | | - | |
248 | | - | |
249 | | - | |
250 | | - | |
251 | | - | |
252 | | - | |
253 | | - | |
254 | | - | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
255 | 278 | | |
| 279 | + | |
| 280 | + | |
256 | 281 | | |
257 | | - | |
258 | | - | |
259 | | - | |
260 | | - | |
261 | | - | |
262 | | - | |
263 | | - | |
264 | | - | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
265 | 291 | | |
266 | 292 | | |
267 | 293 | | |
| |||
272 | 298 | | |
273 | 299 | | |
274 | 300 | | |
275 | | - | |
| 301 | + | |
276 | 302 | | |
277 | 303 | | |
278 | 304 | | |
| |||
289 | 315 | | |
290 | 316 | | |
291 | 317 | | |
292 | | - | |
| 318 | + | |
293 | 319 | | |
294 | 320 | | |
295 | 321 | | |
296 | 322 | | |
297 | 323 | | |
298 | 324 | | |
299 | 325 | | |
300 | | - | |
| 326 | + | |
301 | 327 | | |
302 | 328 | | |
303 | 329 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
20 | | - | |
21 | | - | |
22 | | - | |
23 | | - | |
| 20 | + | |
| 21 | + | |
24 | 22 | | |
25 | | - | |
| 23 | + | |
26 | 24 | | |
27 | | - | |
28 | 25 | | |
29 | 26 | | |
| 27 | + | |
30 | 28 | | |
31 | 29 | | |
32 | 30 | | |
33 | 31 | | |
| 32 | + | |
34 | 33 | | |
35 | 34 | | |
36 | 35 | | |
| |||
119 | 118 | | |
120 | 119 | | |
121 | 120 | | |
122 | | - | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
123 | 126 | | |
124 | 127 | | |
125 | 128 | | |
| |||
149 | 152 | | |
150 | 153 | | |
151 | 154 | | |
152 | | - | |
153 | | - | |
154 | | - | |
155 | | - | |
156 | | - | |
157 | | - | |
158 | | - | |
159 | | - | |
160 | | - | |
161 | | - | |
162 | | - | |
163 | | - | |
164 | | - | |
165 | | - | |
166 | | - | |
167 | | - | |
168 | | - | |
169 | | - | |
170 | | - | |
171 | | - | |
172 | | - | |
173 | | - | |
174 | | - | |
175 | | - | |
176 | | - | |
177 | | - | |
178 | | - | |
179 | | - | |
180 | | - | |
181 | | - | |
182 | | - | |
183 | | - | |
184 | | - | |
185 | | - | |
186 | | - | |
187 | | - | |
188 | | - | |
189 | | - | |
190 | | - | |
191 | | - | |
192 | | - | |
193 | | - | |
194 | | - | |
195 | | - | |
196 | | - | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
197 | 166 | | |
198 | | - | |
199 | | - | |
200 | | - | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
201 | 170 | | |
202 | 171 | | |
203 | 172 | | |
204 | 173 | | |
205 | 174 | | |
206 | 175 | | |
207 | | - | |
208 | | - | |
209 | | - | |
210 | | - | |
211 | | - | |
212 | | - | |
213 | | - | |
214 | | - | |
215 | | - | |
216 | | - | |
217 | | - | |
218 | | - | |
219 | | - | |
220 | | - | |
221 | | - | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
222 | 182 | | |
223 | | - | |
224 | | - | |
225 | 183 | | |
226 | 184 | | |
227 | 185 | | |
228 | 186 | | |
229 | 187 | | |
230 | | - | |
| 188 | + | |
231 | 189 | | |
232 | 190 | | |
233 | 191 | | |
| |||
299 | 257 | | |
300 | 258 | | |
301 | 259 | | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
302 | 278 | | |
303 | 279 | | |
304 | 280 | | |
| |||
0 commit comments