Fix index-oriented JSON parsing

[nitishchauhan002]

Summary

Fixes incorrect parsing of index-oriented JSON in loaders_extra.py.

Type

• Audit
• Explainer
• Bug fix
• Other

---

Audit checklist

• I opened or linked a corresponding issue first
• The folder is named after the domain, not the dataset
• unfair.py includes protected attributes and prints the required output format
• fair.py removes protected attributes and identified proxy variables
• Both scripts use random_state=42 and an 80/20 train/test split
• Proxy variables were actually tested, not just guessed
• unfair.png and fair.png are included as PNG screenshots
• The dataset is public and accessible without login or payment
• The dataset file is included, or DATA.md is included if the file is too large
• README.md includes the new results row and audit section
• A notebook was added if the audit benefits from one

Before fairness gap:

After fairness gap:

Reduction:

Protected attribute(s):

Proxy variables dropped:

---

Explainer checklist

• The file is in explainers/ and uses lowercase hyphenated naming
• It includes a plain-language definition
• It uses a real example from this repo or a documented real-world case
• It includes runnable Python detection or measurement code
• It acknowledges limitations or trade-offs
• It links to related explainers or repo projects
• It includes 2-3 primary sources
• The Explainers table in README.md was updated

---

Linked issue

Closes #405

[vercel]

@nitishchauhan002 is attempting to deploy a commit to the yashkewlani2020-gmailcom's projects Team on Vercel.

A member of the Team first needs to authorize it.

[github-actions]

Hey @nitishchauhan002, your first PR on Fair Code, that's awesome, thank you.

This project is about making AI more accountable, and contributions like yours are what keep that work going. We'll review your changes shortly.

If you haven't already, give the contributing guide a quick read: it covers how audits are structured and what we look for in a review.

And if you're finding Fair Code useful or interesting, a star on the repo genuinely helps more people find it - no pressure, just appreciated.

[github-actions]

@yakew7 @ahmdkaml - new PR touching a path you own, please review.
@Shreyash0712 - new PR

[Copilot]

🟡 Changes recommended

The new dict-of-dicts heuristic can incorrectly transpose valid columns-oriented JSON with a non-default index, creating a new silent wrong-shape failure mode.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR addresses #405 by updating faircode/loaders_extra.py to avoid silently mis-parsing pandas orient="index" JSON (which otherwise loads transposed under pandas’ default "columns" assumption), and adds a targeted regression test for the index-orient case.

Changes:

• Add shape-based JSON-orientation detection for pandas split-orient and index-orient inputs in read_table().
• Add a new test case to ensure index-oriented JSON round-trips without transposition.

File summaries

File	Description
faircode/loaders_extra.py	Adds split-orient detection and a new dict-of-dicts heuristic intended to detect index-oriented JSON before falling back to pandas’ default JSON loader.
tests/test_json_edge_cases.py	Adds a regression test covering pandas orient="index" JSON round-tripping via read_table().

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

The new dict-of-dicts heuristic will misclassify orient="columns" JSON with a non-default index as index-oriented, producing the exact transpose the new test is meant to prevent.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

faircode/loaders_extra.py:69

• The current index-oriented JSON detection will also match pandas orient="columns" output when the DataFrame has a non-default index (inner keys like "a", "b"). In that case read_table() returns pd.read_json(..., orient="index"), which transposes the data and contradicts the new test case below.

        if isinstance(parsed, dict) and parsed and all(
            isinstance(value, dict) for value in parsed.values()
        ):
            inner_key_sets = [set(row.keys()) for row in parsed.values()]

• Files reviewed: 2/2 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

The new index-oriented JSON heuristic in faircode/loaders_extra.py appears to misclassify orient="columns" dict-of-dicts with non-default indices, causing transposed DataFrames and likely failing the added test.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

The current non-square dict-of-dicts heuristic can still silently mis-parse wide orient="index" JSON (rows < columns), leaving the original transpose bug in common cases.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 2
• Review effort level: Lite

[yakew7]

Reviewed this locally (checked out the branch, ran the suite, and built some cases beyond the two included tests) - I don't think this actually resolves the ambiguity, it just moves where the silent data corruption happens. Two concrete repros:

1. A genuinely columns-oriented file with more columns than rows gets silently transposed. A normal 5-column/3-row dataset (5 demographic fields, 3 records), written with orient="columns":

original shape: (3, 5)
loaded shape:   (5, 3)   <- wrong, transposed

No error, no warning - just wrong data. This is the same failure mode the PR is meant to fix, just triggered from the other direction (the "which dimension is larger" heuristic assumes rows > columns, which isn't guaranteed).

2. A genuinely index-oriented, square, all-string dataset also gets silently transposed. A 3x3 sex/race/region dataset (all strings - the normal shape for demographic columns in this repo's own audits), written with orient="index":

original columns: ['sex', 'race', 'region']
loaded columns:   ['p1', 'p2', 'p3']   <- wrong, transposed

The type-homogeneity tie-break only works when columns/rows differ in type mix (one string column + one int column, which is exactly what both included tests use). When every column is a string - the norm for this kind of data - there's nothing to break the tie and it silently defaults to the wrong orientation.

Both new tests pass only because they reuse the same coincidental 2-column, mixed-type, 2-row example, not because the ambiguity is actually resolved. For a fairness-auditing tool, silently transposing/mislabeling demographic columns is worse than the original bug - it can quietly flip which group looks disadvantaged downstream with no signal anything went wrong.

Smaller things while I'm in here:

• Both changed files are missing a trailing newline.
• The get_xlsx_sheet_info edit (blank line + newline removal) looks unrelated to this fix.
• Neither new test exercises the non-square (outer_n != inner_n) branch, or a wide-dataset/homogeneous-column case like the two above.

Suggest replacing the heuristic with what the original issue asked for: detect the ambiguous shape and raise a clear "ambiguous JSON orientation, please specify explicitly" error instead of guessing - a loud, wrong-looking failure is much safer here than a silent one. orient="split" is already handled correctly and round-trips reliably, so that's the format to point people at.

Existing suite + ruff both pass clean on this branch, for what it's worth - the risk is entirely in shapes the current tests don't cover.

[yakew7]

@nitishchauhan002