Where: faircode/proxy.py's proxy_hints(df, dimensions, alpha).
The gap: proxy_hints()" only tests chi-squared independence over pairs drawn from dimensions" — the columns `detect_columns()" found in the dataframe it's given:
cols = [(d["name"], d["kind"]) for d in dimensions]
for i in range(len(cols)):
for j in range(i + 1, len(cols)):
...If a protected attribute (say, age or race) has already been dropped from the dataset entirely — a common, well-intentioned but naive attempt at "fixing" bias by removing the sensitive column — that column can never be one half of a tested pair, because it isn't in df and therefore never makes it into dimensions. `proxy_hints()" will report "no significant associations found" (or simply omit the column) with nothing to distinguish that from a genuinely clean dataset.
Why it matters: this is the textbook failure mode this feature exists to catch — "we dropped the column so it's fine" — and right now the tool goes silent on exactly the case where someone is most likely to reach for it. faircode/SPEC.md's proxy-hints section (§3, lines ~102-108) doesn't mention this limitation either, so a user has no signal that dropping the column removed it from the analysis instead of clearing it.
Suggested fix: extend proxy_hints()" (still opt-in, CLI/Python-only, doesn't touch the frozen score/JS parity per its existing docstring) to optionally accept a **held-out column** — e.g. a held_out={"age": <Series aligned by index/id to df>}" — so a caller can test "does anything remaining in this dataset still predict the column I removed," without requiring the dropped column back in the profiled dataframe itself. At minimum, document the current blind spot explicitly in `faircode/SPEC.md" §3 so it isn't mistaken for "no proxy risk found."
Where:
faircode/proxy.py'sproxy_hints(df, dimensions, alpha).The gap:
proxy_hints()" only tests chi-squared independence over pairs drawn fromdimensions" — the columns `detect_columns()" found in the dataframe it's given:If a protected attribute (say,
ageorrace) has already been dropped from the dataset entirely — a common, well-intentioned but naive attempt at "fixing" bias by removing the sensitive column — that column can never be one half of a tested pair, because it isn't indfand therefore never makes it intodimensions. `proxy_hints()" will report "no significant associations found" (or simply omit the column) with nothing to distinguish that from a genuinely clean dataset.Why it matters: this is the textbook failure mode this feature exists to catch — "we dropped the column so it's fine" — and right now the tool goes silent on exactly the case where someone is most likely to reach for it.
faircode/SPEC.md's proxy-hints section (§3, lines ~102-108) doesn't mention this limitation either, so a user has no signal that dropping the column removed it from the analysis instead of clearing it.Suggested fix: extend
proxy_hints()" (still opt-in, CLI/Python-only, doesn't touch the frozen score/JS parity per its existing docstring) to optionally accept a **held-out column** — e.g. aheld_out={"age": <Series aligned by index/id to df>}" — so a caller can test "does anything remaining in this dataset still predict the column I removed," without requiring the dropped column back in the profiled dataframe itself. At minimum, document the current blind spot explicitly in `faircode/SPEC.md" §3 so it isn't mistaken for "no proxy risk found."