Skip to content

Cached dataframe diff never detects unchanged columns, and lists_are_equal only checks the last element #378

Description

@nikhilsi

Two related comparison bugs in the caching stack at commit 7952236. Filing them
together since both live in the cache diff/signature machinery. (The malformed
clear_cache path templates in the same file already have a fix in open PR #364;
this issue covers the two remaining problems.)

  1. In write_cached_df (openavmkit/utilities/cache.py, lines 296-310), the
    column-modified test is unsatisfiable:
values_equal = col_new.values == col_orig.values
na_equal = col_new.isna() & col_orig.isna()
count_na_equal = na_equal.sum()
count_values_equal = values_equal.sum()
count_to_match = len(col_new)
all_equal = (
    count_na_equal == count_to_match
    and count_values_equal == count_to_match
)

A NaN position can never count as values-equal and a non-NaN position can never
count as na-equal, so the two counts cannot both reach len(col_new) for any
nonempty column. Every common column is therefore flagged as modified and
rewritten on every save, defeating the incremental diff. Fix: a position
matches if it is values-equal OR both-NaN:

all_equal = bool((values_equal | na_equal.to_numpy()).sum() == len(col_new))
  1. lists_are_equal (openavmkit/utilities/assertions.py, lines 104-107)
    overwrites result on every iteration, so only the last element decides:
for i in range(len(a)):
    entry_a = a[i]
    entry_b = b[i]
    result = objects_are_equal(entry_a, entry_b)

lists_are_equal([2, 2], [3, 2]) returns True. This helper backs
dicts_are_equal, which the cache uses for signature matching (cache.py line
510), and several test assertions. Fix: break with False on the first mismatch.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions