Skip to content

manifest.py: numeric_threshold/age_interval_threshold with no 'threshold' field silently classifies zero rows #415

Description

@yakew7

Where: faircode/manifest.py's ProtectedAttribute dataclass - threshold: float | None = None has no required-ness validation for the numeric_threshold/age_interval_threshold types that depend on it.

Repro:

>>> from faircode.manifest import ProtectedAttribute
>>> import pandas as pd
>>> pa = ProtectedAttribute(name="age", type="numeric_threshold", column="age")  # no threshold
>>> pa.disadvantaged_mask(pd.DataFrame({"age": [20, 30, 40]}))
(0    False
 1    False
 2    False
 dtype: bool, 0    True
 1    True
 2    True
 dtype: bool)

numeric < None evaluates to all-False in pandas rather than raising, so a manifest with a missing threshold field silently produces zero disadvantaged rows for that attribute, and known incorrectly reports every row as classifiable.

Why it matters: this doesn't crash - it fails several steps downstream, where compute_metrics sees an all-False disadvantaged mask and reports "insufficient_data". A contributor debugging a confusing "insufficient data" result would have to trace back through several layers to discover the actual problem is a missing threshold field in their audit.yaml, not genuinely sparse data.

Suggested fix: raise a clear error (e.g. in ProtectedAttribute.__post_init__ or wherever manifests are loaded from YAML) when type is numeric_threshold/age_interval_threshold and threshold is None, naming the attribute and the missing field - mirroring the existing "need disadvantaged_values or advantaged_values" check already present for the categorical type.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinghelp wantedExtra attention is needed

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions