Description
#6236 (fixing #5868) makes filters treat -0.0 and 0.0 as the same value when one side is a literal. Comparisons between two float values, and array_has with a zero, still compare by encoding:
Steps to reproduce
import lance
import pyarrow as pa
data = pa.table({
"id": [0, 1, 2],
"a": [-0.0, 0.0, 1.0],
"b": [0.0, -0.0, 1.0],
"l": [[-0.0], [0.0], [1.0]],
})
ds = lance.write_dataset(data, "zero.lance", mode="overwrite")
def ids(filter):
return sorted(ds.to_table(columns=["id"], filter=filter)["id"].to_pylist())
print("a = 0.0 ", ids("a = 0.0")) # fixed by #6236
print("a = b ", ids("a = b"))
print("a < b ", ids("a < b"))
print("array_has(l, 0.0) ", ids("array_has(l, 0.0)"))
print("array_has(l, -0.0) ", ids("array_has(l, -0.0)"))On pylance 13.0.0b4:
a = 0.0 [0, 1]
a = b [2]
a < b [0]
array_has(l, 0.0) [1]
array_has(l, -0.0) [0]
Expected behavior
a = b [0, 1, 2]
a < b []
array_has(l, 0.0) [0, 1]
array_has(l, -0.0) [0, 1]
Lance version
13.0.0b4
Language binding
Python
Logs / traceback
Description
#6236 (fixing #5868) makes filters treat
-0.0and0.0as the same value when one side is a literal. Comparisons between two float values, andarray_haswith a zero, still compare by encoding:Steps to reproduce
On pylance 13.0.0b4:
Expected behavior
Lance version
13.0.0b4
Language binding
Python
Logs / traceback
INlists; the scan filter here still goes through Arrow's total-order kernels.array_hascould be rewritten the same way as0.0 IN (a, b)is in fix: make float filters treat -0.0 and 0.0 as the same value #6236:array_has(l, 0.0)→array_has(l, -0.0) OR array_has(l, 0.0).-1.0 * 0.0produces-0.0in ordinary analytics, which makes column-to-column cases likea * -1 = brealistic.