Skip to content

flaky: dash/TestSpendSurvivesRowEviction is red for the first 10 hours of every month (fixture backdates across the month boundary) #157

Description

@amiddavid

dash/TestSpendSurvivesRowEviction fails deterministically, not intermittently, for the first
10 hours of every calendar month. It is failing right now on main (e88f5d8).

=== RUN   TestSpendSurvivesRowEviction
    spend_test.go:43: month-to-date = 7, want 10
--- FAIL: TestSpendSurvivesRowEviction (0.03s)

Cause

spendEvents (dash/spend_test.go:12) backdates its fixture by one hour per session:

base := time.Now().Add(-time.Duration(sessions) * time.Hour).UnixMilli()

The test asks for 10 sessions, so the oldest event is 10 hours old, and then asserts
MonthToDateUSD("t1") == 10.0 — i.e. that all 40 priced requests fall inside the current
month. Run at 07:42 UTC on the 1st, 3 of the 10 sessions land on the previous month's last
day, MonthToDateUSD correctly excludes them, and the assertion fails with 7.

Nothing is wrong with MonthToDateUSD or with DropOldestSessionsOfTenant — the fixture is
asserting a property of the clock. It is 3/3 reproducible on main at
e88f5d8, on the current time; it will pass again after 10:00 UTC today.

Severity

Medium. The suite gates every deploy, and this is a hard red for a ~10-hour window each month
(and for the whole of a shorter window in other timezones' CI). It is distinct from the two
load-dependent flakes in #91: those need a saturated box, this one needs only a date.

Fix options

  1. Inject the clock into the fixture — build the events from a fixed timestamp that is
    known to be inside the month, e.g. anchor at the 15th of the current month rather than at
    time.Now(). Cheapest, keeps the property being tested intact. Recommended.
  2. Assert relative rather than absolute — read MonthToDateUSD before eviction and only
    assert it does not fall afterwards, dropping the == 10.0 precondition. Loses the check
    that the fixture priced what it thought it priced, and that precondition is what makes the
    "did not fall" assertion non-vacuous, so this trades away real coverage.
  3. Shrink the backdating — spread the sessions over minutes instead of hours. Narrows the
    window to ~10 minutes a month rather than closing it, and the sessions must stay distinct
    in time for the eviction ordering, so this only moves the bug.

Found while running the full suite to verify PR #138 (which touches neither dash Go code nor
spend accounting); reproduced on unmodified main to confirm it is pre-existing.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions