Where: faircode/significance.py:159 (intersectional_report, called from faircode/benchmark.py:165-166).
The gap: intersectional_report calls significance_report(y[both], y[neither], ...) unconditionally, with no check that the both/neither intersectional cells are non-empty. The function's own _rate() helper does guard against an empty mask (if mask.any() else float("nan")) - the same guard was never applied before the significance_report call two lines above it.
Repro:
>>> import numpy as np, warnings
>>> from faircode.significance import significance_report
>>> with warnings.catch_warnings():
... warnings.simplefilter("ignore")
... significance_report(np.array([]), np.array([1,0,1,0,1]), n_resamples=100, n_permutations=100)
{'gap': nan, 'ci_low': nan, 'ci_high': nan, 'p_value': 0.0, 'significant': True, 'n_a': 0, 'n_b': 5, 'small_sample_warning': True}
An empty group produces gap: nan alongside p_value: 0.0 and significant: True - a statistically meaningless result reported as a confirmed finding.
Why it matters: this flows straight into benchmark.py's fairness_rows for any audit with 2+ protected attributes where a pairwise intersectional cell happens to be empty on a filtered subgroup (3 of the 7 shipped audits declare 3+ protected attributes, making this plausible). A benchmark run could report a statistically significant intersectional gap that's actually undefined - the opposite of a false negative, actively fabricating a finding. tests/test_significance.py only covers non-empty cells.
Suggested fix (in the caller, since significance.py is frozen per CLAUDE.md): in benchmark.py's run_audit, check cell_sizes["both"] == 0 or cell_sizes["neither"] == 0 (available from intersectional_report's own return value) before trusting the intersectional row's significant/p_value, and mark/exclude it instead.
Where:
faircode/significance.py:159(intersectional_report, called fromfaircode/benchmark.py:165-166).The gap:
intersectional_reportcallssignificance_report(y[both], y[neither], ...)unconditionally, with no check that theboth/neitherintersectional cells are non-empty. The function's own_rate()helper does guard against an empty mask (if mask.any() else float("nan")) - the same guard was never applied before thesignificance_reportcall two lines above it.Repro:
An empty group produces
gap: nanalongsidep_value: 0.0andsignificant: True- a statistically meaningless result reported as a confirmed finding.Why it matters: this flows straight into
benchmark.py'sfairness_rowsfor any audit with 2+ protected attributes where a pairwise intersectional cell happens to be empty on a filtered subgroup (3 of the 7 shipped audits declare 3+ protected attributes, making this plausible). A benchmark run could report a statistically significant intersectional gap that's actually undefined - the opposite of a false negative, actively fabricating a finding.tests/test_significance.pyonly covers non-empty cells.Suggested fix (in the caller, since
significance.pyis frozen per CLAUDE.md): inbenchmark.py'srun_audit, checkcell_sizes["both"] == 0 or cell_sizes["neither"] == 0(available fromintersectional_report's own return value) before trusting the intersectional row'ssignificant/p_value, and mark/exclude it instead.