Skip to content

Fill missing categoricals as UNKNOWN instead of the string "nan" - #394

Open
nikhilsi wants to merge 1 commit into
larsiusprime:masterfrom
nikhilsi:fix-categorical-fill-nan
Open

Fill missing categoricals as UNKNOWN instead of the string "nan"#394
nikhilsi wants to merge 1 commit into
larsiusprime:masterfrom
nikhilsi:fix-categorical-fill-nan

Conversation

@nikhilsi

@nikhilsi nikhilsi commented Jul 4, 2026

Copy link
Copy Markdown

What was wrong

In openavmkit/cleaning.py, _fill_unknown_values (lines 748-749 at
7952236) casts categorical fields to str before filling:

df[field] = df[field].astype("str")
df[field] = df[field].fillna("UNKNOWN")

astype("str") converts NaN to the literal string "nan" first, so the
fillna on the next line never finds anything to fill. Models end up with
a "nan" category instead of the intended "UNKNOWN".

The change

Reorder so the fill happens while the nulls are still real:

df[field] = df[field].astype("object").fillna("UNKNOWN").astype("str")

Going through object keeps NaN intact for the fill and also sidesteps the
categorical-dtype restriction on filling with an unseen category.

What changes downstream

The category label for missing values changes from "nan" to "UNKNOWN", so
one-hot column names and tree categorical levels shift accordingly.
Refitting after upgrading will produce slightly different encodings than
cached models built against the "nan" label; model caches should be
invalidated once this lands.

Verification

A test is included. It runs a frame containing NaN in a declared
categorical field through the fill step and asserts the result contains
"UNKNOWN" and no "nan" values. At 7952236 the "nan" assertion fails; with
this change it passes.

Fixes #377.

astype("str") converts NaN to the literal string "nan" before
fillna runs, so models received a "nan" category instead of
"UNKNOWN". Fill on object dtype first, then cast. Adds
tests/test_cleaning.py covering the null-to-UNKNOWN path.
@github-actions

github-actions Bot commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution.
Please sign our CLA at the following link:
Click here to sign the CLA.

No action is required from you in this PR thread. Once you have signed the CLA externally, a maintainer will verify your signature and record it here on your behalf by commenting:


I affirm that this contributor has signed the CLA


You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Categorical fill produces a literal "nan" category instead of "UNKNOWN"

1 participant