Introduce Chatterjeexi as standard correlation for feature selection

[thomasATbayer]

Why Chatterjee's Xi for Feature Selection

Chatterjee's Xi is superior for feature selection in heterogeneous feature sets because:

1. Non-linear Relationship Detection

• Pearson only captures linear relationships (correlation = slope of best-fit line)
• Chatterjee's Xi detects any monotonic or functional relationship, including:
  • Quadratic: y = x²
  • Exponential: y = e^x
  • Logarithmic: y = log(x)
  • Step functions and V-shapes

2. Scale Invariance

• Works equally well across features with different scales and distributions
• No assumption about normality or specific distributions
• Handles mixed feature types (continuous, discrete, bounded)

3. Robustness to Outliers

• Based on rank statistics rather than raw values
• Less sensitive to extreme values than Pearson
• Better for chemical/molecular features with occasional extreme values

4. Asymmetric Dependencies

• Measures "can Y be explained by X?" (not just association)
• Perfect for feature selection: "can this feature explain the target?"
• Detects one-directional predictive relationships

5. Heterogeneous Feature Sets

In molecular/chemical datasets, you typically have:

• Continuous: molecular weight, logP
• Discrete: ring counts, atom counts
• Binary: presence/absence of functional groups
• Bounded: TPSA (0-200), Lipinski descriptors

Chatterjee's xi handles all these uniformly without requiring feature-specific preprocessing.

[thomasATbayer]

def chatterjeexi_correlation(x: Union[npt.ArrayLike, pd.Series], y: Union[npt.ArrayLike, pd.Series]) -> float:
"""Wrapper for Chatterjee's xi correlation with special handling for edge cases.

Chatterjee's xi is a measure of dependence that is more robust than Pearson
for non-linear relationships. This wrapper handles edge cases to make it
suitable for feature selection:
- NaN values (constant features) are mapped to 1.0 to ensure removal
- When X is constant, also return 1.0 for consistency across comparison orders
- Scale is reversed (1 - xi) so that high values indicate high correlationExpand commentComment on line R58Resolved

Parameters:
    x (Union[npt.ArrayLike, pd.Series]): First variable (array-like or pandas Series).
    y (Union[npt.ArrayLike, pd.Series]): Second variable (array-like or pandas Series).

Returns:
    float: Modified correlation coefficient (0 to 1, where 1 = perfect correlation).
"""
import numpy as npExpand commentComment on line R67Resolved

x_arr = np.asarray(x)
y_arr = np.asarray(y)

# Use min/max comparison for O(n) constant detection (faster than np.unique which is O(n log n))
x_is_constant = np.min(x_arr) == np.max(x_arr)
y_is_constant = np.min(y_arr) == np.max(y_arr)

# Check if BOTH features are constant - these are true duplicates
if x_is_constant and y_is_constant:
    return 1.0

# If X is constant but Y varies: constant cannot explain varying, return 0
# This ensures varying features are never dropped at any threshold
if x_is_constant and not y_is_constant:
    return 0.0

result = scipy_stats.chatterjeexi(x_arr, y_arr)
statistic = result.statistic

# Handle NaN case (Y is constant but X varies)
# Return 1.0: constant Y can always be "explained" (it's perfectly predictable)
# SmartCorrelatedSelection will drop the constant (based on selection_method)
if np.isnan(statistic):
    return 1.0

# Chatterjee's xi is already in [0, 1] range where higher values indicate stronger dependence
# However, it can occasionally be negative for random/uncorrelated data due to finite sample effects
# Clamp to [0, 1] range for feature selection
return np.clip(statistic, 0.0, 1.0)

[thomasATbayer]

Also check if both are identical, as even with a small sample size hould be 1 then, chatterjeexy is smaple size dependent