Repository navigation
[C++][Parquet] Be better to check null_count, distinct_count and nan_count are negative when reading files #50692
Description
Activity
Hi @HuaHuaY,
I'd like to work on this enhancement if it's available.
From my understanding, the change involves validating that
null_countanddistinct_countare not negative while reading Parquet metadata, and handling invalid values appropriately instead of blindly accepting them.I'll first inspect the metadata reading path and existing validation logic to ensure the change is consistent with the current error-handling behavior.
Please let me know if there are any implementation preferences or edge cases you'd like me to consider. Thanks!
I'd like to work on this enhancement if it's available.
Hi @KHARSHAVARDHAN-eng,
This is a minor issue I found when I try to implement #50689. And I have a PR to fix it. If you'd like, you can help to review my PR or take a look at other issues in the repository. Sorry for taking up your time, and thank you for wanting to help fix it.Hi @KHARSHAVARDHAN-eng, You've already opened some PRs in the last couple of days and we have already commented on some of those. They have several issues, could you fix them instead of working on new ones? Thanks.
Close this issue. Referring to #50694 (comment), we will not check whether
xxx_countis negative.
Describe the enhancement requested
In
parquet.thrift,null_count,distinct_count, andnan_countare defined asoptional i64, so it is possible to encounter a negative value (whether due to an incorrect file writer implementation, file corruption, or other reasons). We should check for negative values. However, the Parquet format does not specify whether to throw an error or treat such values as "missing" when encountered; I lean towards the latter, as it avoids blocking the reading of other data.nan_countis not supported now. We can checknull_countanddistinct_countfirst, and checknan_countwhen we implement #50689.Component(s)
C++, Parquet