Where: faircode/manifest.py:103,113 (ProtectedAttribute.disadvantaged_mask, numeric_threshold/age_interval_threshold branches).
The gap:
disadv = numeric < self.threshold if self.disadvantaged == "below" else numeric >= self.threshold
self.disadvantaged is compared against the exact string "below" with no validation anywhere that it's actually one of the two documented values ("below"/"above"). Any other value - "Below", "BELOW", a trailing space, a typo - silently takes the "above" branch instead of raising.
Repro:
>>> from faircode.manifest import ProtectedAttribute
>>> import pandas as pd
>>> pa = ProtectedAttribute(name="age", type="numeric_threshold", column="age", threshold=30, disadvantaged="Below")
>>> pa.disadvantaged_mask(pd.DataFrame({"age": [20, 30, 40]}))[0].tolist()
[False, True, True]Intended (disadvantaged = below 30): [True, False, False]. A single capitalized letter in audit.yaml silently inverts which group the entire audit treats as disadvantaged, with zero error or warning.
Why it matters: this is a hand-written YAML field a contributor types directly (see faircode/MANIFEST_SPEC.md) - exactly the kind of value a typo reaches. For a fairness-auditing tool, silently swapping which group is "disadvantaged" produces a benchmark result that looks complete and plausible while being backwards.
Suggested fix: validate disadvantaged in ("below", "above") in ProtectedAttribute.__post_init__ (or wherever manifests are loaded from YAML), raising a clear error naming the bad value - mirroring the existing raise ValueError(f"unknown protected attribute type: {self.type!r}") pattern two lines below.
Where:
faircode/manifest.py:103,113(ProtectedAttribute.disadvantaged_mask,numeric_threshold/age_interval_thresholdbranches).The gap:
self.disadvantagedis compared against the exact string"below"with no validation anywhere that it's actually one of the two documented values ("below"/"above"). Any other value -"Below","BELOW", a trailing space, a typo - silently takes the"above"branch instead of raising.Repro:
Intended (disadvantaged = below 30):
[True, False, False]. A single capitalized letter inaudit.yamlsilently inverts which group the entire audit treats as disadvantaged, with zero error or warning.Why it matters: this is a hand-written YAML field a contributor types directly (see
faircode/MANIFEST_SPEC.md) - exactly the kind of value a typo reaches. For a fairness-auditing tool, silently swapping which group is "disadvantaged" produces a benchmark result that looks complete and plausible while being backwards.Suggested fix: validate
disadvantaged in ("below", "above")inProtectedAttribute.__post_init__(or wherever manifests are loaded from YAML), raising a clear error naming the bad value - mirroring the existingraise ValueError(f"unknown protected attribute type: {self.type!r}")pattern two lines below.