Add explainers/multiple-comparisons.md.
Bootstrap Confidence Intervals already names this exact gap in its own Limitations section: "This repo's benchmark harness computes six metrics across every protected attribute, model, and strategy for seven audits - dozens of p-values per audit. At a 0.05 threshold, running enough independent tests will eventually produce a 'significant' result by chance alone... No correction (e.g. Bonferroni) is currently applied across this full set." That's a self-flagged, real gap worth its own explainer rather than a single limitations paragraph - the actual mechanics of why dozens of p-values inflate false-positive risk, and what a correction (Bonferroni, Holm, Benjamini-Hochberg FDR) actually does about it, isn't covered anywhere yet.
Suggested structure (match bootstrap-confidence-intervals.md): one-sentence definition -> why it matters (a repo-scale example: six metrics x N attributes x N models x 5 strategies x 7 audits = dozens of independent p-values per audit run) -> core concept (family-wise error rate vs. false discovery rate; what Bonferroni/Holm/BH-FDR each actually do to a p-value threshold) -> concrete example applying a correction to a real slice of paper/results-frozen/results_fairness.csv's p-values, showing which "significant" results survive correction and which don't -> runnable code implementing at least one correction method from scratch -> limitations (a correction trades false positives for false negatives - it can hide a real, smaller gap) -> related concepts/projects/further reading.
Related: bootstrap-confidence-intervals, base-rate-fallacy, intersectional-bias.
See CONTRIBUTING.md: add explainers/<slug>.md plus a one-line entry in assets/explainers-data.json; the build script generates the page.
Freeze-safe: explainers are fully open for contribution during the paper freeze. If you quote any Fair Code benchmark result, use the frozen numbers in paper/results-frozen/ (never re-run your own) - see CLAUDE.md.
Add
explainers/multiple-comparisons.md.Bootstrap Confidence Intervals already names this exact gap in its own Limitations section: "This repo's benchmark harness computes six metrics across every protected attribute, model, and strategy for seven audits - dozens of p-values per audit. At a 0.05 threshold, running enough independent tests will eventually produce a 'significant' result by chance alone... No correction (e.g. Bonferroni) is currently applied across this full set." That's a self-flagged, real gap worth its own explainer rather than a single limitations paragraph - the actual mechanics of why dozens of p-values inflate false-positive risk, and what a correction (Bonferroni, Holm, Benjamini-Hochberg FDR) actually does about it, isn't covered anywhere yet.
Suggested structure (match bootstrap-confidence-intervals.md): one-sentence definition -> why it matters (a repo-scale example: six metrics x N attributes x N models x 5 strategies x 7 audits = dozens of independent p-values per audit run) -> core concept (family-wise error rate vs. false discovery rate; what Bonferroni/Holm/BH-FDR each actually do to a p-value threshold) -> concrete example applying a correction to a real slice of
paper/results-frozen/results_fairness.csv's p-values, showing which "significant" results survive correction and which don't -> runnable code implementing at least one correction method from scratch -> limitations (a correction trades false positives for false negatives - it can hide a real, smaller gap) -> related concepts/projects/further reading.Related: bootstrap-confidence-intervals, base-rate-fallacy, intersectional-bias.
See CONTRIBUTING.md: add
explainers/<slug>.mdplus a one-line entry inassets/explainers-data.json; the build script generates the page.