Skip to content

bug: Selector union (|) returns columns in a different order on non-Polars backends #4029

Open

Description

@MarcoGorelli

(from claude)

When you combine two selectors with |, Polars returns the columns in schema order. The other backends return the left selector's columns first, then the right selector's.

import duckdb
import pandas as pd
import polars as pl
import pyarrow as pa

import narwhals as nw
import narwhals.selectors as ncs

data = {"a": [1], "b": ["x"], "c": [1.0], "d": [True]}

for native in (pl.DataFrame(data), pd.DataFrame(data), pa.table(data)):
    print(nw.from_native(native).select(ncs.boolean() | ncs.numeric()).columns)

rel = duckdb.sql("select 1 as a, 'x' as b, 1.0::double as c, true as d")
print(nw.from_native(rel).select(ncs.boolean() | ncs.numeric()).columns)

Output:

['a', 'c', 'd']  # polars
['d', 'a', 'c']  # pandas
['d', 'a', 'c']  # pyarrow
['d', 'a', 'c']  # duckdb

Native Polars (pl.selectors.boolean() | pl.selectors.numeric()) gives ['a', 'c', 'd']. Since Narwhals follows the Polars API, the expected result on every backend is ['a', 'c', 'd'].

Cause

CompliantSelector.__or__ in src/narwhals/_compliant/selectors.py returns the left selector's columns, followed by any right-selector columns not already included. It never puts them back into schema order. & and - don't have this problem because they keep the left selector's order, which is already schema order.

Suggested fix

After combining the two sides, filter the frame's schema by membership in the result, so columns come out in schema order. This should land together with #3621, which adds ^ with the same pattern and the same problem. Tests should include a case where the left selector's columns come after the right selector's in the schema, as in the example above. The current tests miss this because all their cases are already in schema order.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bug: incorrect resultSomething isn't workinghigh priorityYour PR will be reviewed very quickly if you address this

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions