Fix encoded_missing_value being ignored and all-missing datetime columns crashing - #76
devYRPauli wants to merge 3 commits into
Conversation
sylvesterkaczmarek
left a comment
There was a problem hiding this comment.
transform() now emits encoded_missing_value, but inverse_transform() still only excludes unknown_value from the category lookup. With the example encoded_missing_value=-99, a missing value therefore reaches cats[-99], which can raise or decode to an unrelated category. Should inverse_transform() handle the missing sentinel explicitly as well?
transform() now writes encoded_missing_value for NaN inputs, so inverse_transform() can receive that sentinel. Its mask only excluded unknown_value, so the sentinel reached cats[value]. With encoded_missing_value=-99 and three categories that raises IndexError; with 120 categories it returns cats[-99], an unrelated category, and raises nothing. Exclude the missing sentinel from the same mask. Missing values now decode to None, matching how unknown values already decode and what main returns today. The defaults and both in-repo construction sites use -1 for both sentinels, so their behavior is unchanged.
|
You are right. I measured both halves of it. Before this PR Measured with
So both failure modes you named are real. The second one is the worse of the two, because it decodes to an unrelated category and raises nothing. Reach inside the repo is narrower. Both construction sites pass Fixed by excluding the missing sentinel from the same mask: valid_mask = (col != self.unknown_value) & (
col != self.encoded_missing_value
)Missing now decodes to Added |
Two independent edge case fixes in
tabfm/src/classifier_and_regressor.py, one per commit.1.
CategoricalOrdinalEncoderignoresencoded_missing_valueencoded_missing_valueis accepted in__init__and documented as "Encoded value used for NaN / missing values", buttransform()never reads it. Missing values and unseen categories are both filled withunknown_value, so the parameter has no effect.The fix records the original NaN positions before mapping and fills those with
encoded_missing_value, leavingunknown_valuefor values genuinely absent from the category map. Both parameters default to-1, so behaviour is unchanged unless a caller sets them differently.2.
DatetimeTransformercrashes on an all-missing datetime columnfit()storesseries.mean()as the fill value. For a column with no valid timestamps that mean isNaT, so the laterfillnais a no-op and the subsequent.astype(np.int64)on the extracted parts raises:The fix falls back to the epoch when the computed mean is
NaT. Columns with at least one valid timestamp are unaffected.Tests
Added
CategoricalOrdinalEncoderTestandDatetimeTransformerTesttoclassifier_and_regressor_test.py. Both were confirmed to fail against the unpatched source (-1.0vs-99for the first,IntCastingNaNErrorfor the second) and pass with the change.Full existing suite after the change: 48 tests run, 20 passed, 28 skipped because JAX is not installed in my environment. Those 28 remain unverified locally.
These changes do not touch any line modified by my open PRs #59 or #69.