bug: a NaN with its sign bit set sorts below every number in filters #9315

Description

Float filters order values by IEEE 754 total order, where a NaN with its sign bit set comes before -inf. SQL engines (PostgreSQL treats every NaN as equal and greater than any number), IEEE 754 comparisons, and dataframe libraries such as Polars don't order NaN that way. A negative NaN is not unusual either: on x86 it is the NaN produced by 0.0 / 0.0, inf - inf, sqrt(-1) and inf * 0 in numpy, pyarrow and Polars, and by negating a NaN.

Steps to reproduce

So the same logical value answers differently depending on its sign bit:

import math
import lance
import pyarrow as pa

neg_nan = math.copysign(math.nan, -1.0)  # what 0.0/0.0 and inf-inf produce on x86
data = pa.table({"id": [0, 1, 2, 3], "x": [neg_nan, math.nan, 1.0, 2.0]})
ds = lance.write_dataset(data, "nan.lance", mode="overwrite")

def ids(filter):
    return sorted(ds.to_table(columns=["id"], filter=filter)["id"].to_pylist())

print("x > 1.0      ", ids("x > 1.0"))
print("x < 1.0      ", ids("x < 1.0"))
print("isnan(x)     ", ids("isnan(x)"))
ds.create_scalar_index("x", "BTREE")
ds = lance.dataset("nan.lance")
print("x > 1.0 btree", ids("x > 1.0"))

On pylance 13.0.0b4 (same on 11.0.0 and 9.0.0):

x > 1.0       [1, 3]
x < 1.0       [0]
isnan(x)      [0, 1]
x > 1.0 btree [1, 3]

Rows 0 and 1 are both NaN (isnan agrees), yet x > 1.0 keeps only the positive one and x < 1.0 keeps the negative one. A column-to-column comparison is affected the same way: a negative and a positive NaN compare unequal, and the negative one compares less than any number. The BTREE index agrees with the scan, so the result is consistent across plans, just not what the data means.

Expected behavior

The sign of a NaN doesn't affect comparisons: x > 1.0 returns [1, 3] plus row 0, and x < 1.0 returns neither NaN.

Lance version

13.0.0b4

Language binding

Python

Notes

• Same root cause as incorrect handling of -0.0 in comparison #5868: Arrow's comparison kernels use total_cmp and ask callers to normalize. fix: make float filters treat -0.0 and 0.0 as the same value #6236 fixed this for signed zeros by rewriting literals in the planner; NaN signs aren't covered.
• DataFusion's IN list work (IN LIST: treat signed zeros as equal apache/datafusion#25186) deliberately keeps distinct NaN payloads distinct, so this probably has to be handled in Lance, as fix: make float filters treat -0.0 and 0.0 as the same value #6236 did for zeros.
• A literal-side rewrite is possible without losing the index, since x < CAST('-inf' AS double) holds for exactly the negative NaNs: x > c → x > c OR x < -inf, x < c → x < c AND x >= -inf. Normalizing NaN to one sign on write, or in the comparison, would also cover column-to-column comparisons.