Skip to content

monitor: port time-denominated dwell gate to eval_counter_ratio (2x streak acceleration) #3919

Description

@tomerweller

Follow-up from #3790. The time-denominated dwell gate landed in #3790 fixes the 2x streak-acceleration bug in eval_counter_streak (the recovery-stalled alarm). The identical tick-vs-time root cause exists in eval_counter_ratio (scripts/lib/eval-alarms.py): its {name}_streak advances once per breaching tick, so a duplicate/split interval advances the ratio streak at 2x wall-clock speed and reaches streak >= streak_threshold early — same defect, same fix.

Affected alarms (kind = counter-ratio): scp-accept-rate-low, apply-failure-ratio, pending-too-old-ratio.

Fix: port the #3790 pattern to eval_counter_ratio — add a per-alarm {name}_first_breach_ts to the ratio_snapshot, set it on the 0→1 streak open, preserve it while advancing, clear it on every reset branch (low-volume, den==0, non-breaching, gap-stale, too-fresh, counter/PID reset), and gate the fire on now - first_breach_ts >= streak_threshold * expected_interval_seconds in addition to the count. Reuse the expected_interval_seconds config field (default 1200s) introduced in #3790, and plumb now into eval_counter_ratio (already threaded into eval_counter_streak).

Deferred from #3790 to keep that PR atomic on the counter-streak family (the filed incident). See #3757 comment 5105990247 for the root-cause writeup.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions