Skip to content

manifest.py: TargetSpec 'equals'/'isin' without their value field silently mislabels or crashes unhelpfully #416

Description

@yakew7

Where: faircode/manifest.py:58-61 (TargetSpec.compute, "equals"/"isin" methods).

The gap: neither method validates that its required companion field is actually set - value for "equals", values for "isin" - and the two failure modes are inconsistent with each other:

>>> from faircode.manifest import TargetSpec
>>> import pandas as pd
>>> df = pd.DataFrame({"flag": [1, 0, 1]})
>>> TargetSpec(column="flag", method="equals").compute(df).tolist()   # no `value`
[0, 0, 0]
>>> TargetSpec(column="flag", method="isin").compute(df)              # no `values`
TypeError: only list-like objects are allowed to be passed to isin(), you passed a `NoneType`

"equals" with a missing value silently compares every row to None and produces an all-zero label column with no error at all - the model would train on a constant target with no diagnostic anywhere. "isin" with missing values does raise, but with a raw pandas TypeError that never mentions audit.yaml or which manifest field is wrong.

Why it matters: an all-zero (or all-one) target is one of the most silent, hard-to-notice manifest mistakes possible - training/evaluation will "succeed" and produce numbers, just meaningless ones.

Suggested fix: validate the required companion field at TargetSpec construction (or manifest load time), raising a clear error for both "equals" without value and "isin" without values - matching ProtectedAttribute's existing "need disadvantaged_values or advantaged_values" validation style.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinghelp wantedExtra attention is needed

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions