fix: replace manual anomalies with a hampel filter #1997
+155
−353
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,129 +1,90 @@ | ||
| import type { EvolutionData } from '~/types/chart' | ||
|
|
||
| /** | ||
| * Hampel filter for automatic anomaly detection and correction. | ||
| * | ||
| * For each data point, computes the median and Median Absolute Deviation (MAD) | ||
| * of a surrounding window. Points deviating more than `threshold` MADs from | ||
| * the local median are flagged as anomalies and replaced with the median. | ||
| * | ||
| * This approach is unbiased — it applies the same statistical test to every | ||
| * package equally, with no manual curation. | ||
| */ | ||
|
|
||
| const DEFAULT_HALF_WINDOW = 3 | ||
| const DEFAULT_THRESHOLD = 3 | ||
|
|
||
| function median(values: number[]): number { | ||
| const sorted = [...values].sort((a, b) => a - b) | ||
| const mid = Math.floor(sorted.length / 2) | ||
| return sorted.length % 2 !== 0 ? sorted[mid]! : (sorted[mid - 1]! + sorted[mid]!) / 2 | ||
| } | ||
|
|
||
| function mad(values: number[], med: number): number { | ||
| const deviations = values.map(v => Math.abs(v - med)) | ||
| return median(deviations) | ||
| } | ||
|
|
||
| export function applyHampelCorrection( | ||
| data: EvolutionData, | ||
| opts?: { halfWindow?: number; threshold?: number }, | ||
| ): EvolutionData { | ||
| // halfWindow controls how many neighbors on each side to consider. | ||
| // A window of 3 means we look at 7 points total (3 left + current + 3 right). | ||
| const halfWindow = opts?.halfWindow ?? DEFAULT_HALF_WINDOW | ||
|
|
||
| // threshold controls sensitivity. A value of 3 means a point must deviate | ||
| // more than 3 scaled MADs from the local median to be flagged. | ||
| // Higher = less sensitive, lower = more aggressive filtering. | ||
| const threshold = opts?.threshold ?? DEFAULT_THRESHOLD | ||
|
|
||
| // Not enough data to form a full window — return as-is. | ||
| if (data.length < halfWindow * 2 + 1) return data | ||
|
|
||
| const values = (data as Array<{ value: number }>).map(d => d.value) | ||
| // Clone to avoid mutating the original data. | ||
| const result = (data as Array<Record<string, any>>).map(d => ({ ...d })) | ||
|
|
||
| for (let i = 0; i < values.length; i++) { | ||
| // Build a sliding window around the current point, clamped to array bounds. | ||
| const start = Math.max(0, i - halfWindow) | ||
| const end = Math.min(values.length - 1, i + halfWindow) | ||
| const window = values.slice(start, end + 1) | ||
|
|
||
| // The median is robust to outliers — unlike the mean, a single spike | ||
| // won't pull it away from the true central tendency. | ||
| const windowMedian = median(window) | ||
|
|
||
| // MAD (Median Absolute Deviation) measures spread without being | ||
| // influenced by the outliers we're trying to detect. | ||
| const windowMad = mad(window, windowMedian) | ||
|
|
||
| // How far this point is from the local median. | ||
| const deviation = Math.abs(values[i]! - windowMedian) | ||
|
|
||
| // MAD of 0 means most values in the window are identical. | ||
| // If this point differs from the median at all, it's an outlier. | ||
| if (windowMad === 0) { | ||
| if (deviation > 0) { | ||
| result[i]!.value = Math.round(windowMedian) | ||
| result[i]!.hasAnomaly = true | ||
| } | ||
| continue | ||
|
Comment on lines
+65
to
+72
Contributor
There was a problem hiding this comment. The zero-MAD branch will erase real low-volume traffic. When |
||
| } | ||
|
|
||
| // Scale MAD to approximate standard deviation using the consistency | ||
| // constant 1.4826 (valid for normally distributed data). | ||
| // The resulting score is essentially "how many standard deviations | ||
| // away from the local median is this point?" | ||
| const score = deviation / (windowMad * 1.4826) | ||
|
|
||
| // If the score exceeds the threshold, replace with the median. | ||
| // This corrects the spike while preserving the surrounding trend. | ||
| if (score > threshold) { | ||
| result[i]!.value = Math.round(windowMedian) | ||
| result[i]!.hasAnomaly = true | ||
| } | ||
| } | ||
|
|
||
| return result as EvolutionData | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do not score boundary samples with truncated windows.
Line 33 says a
halfWindowof3uses3neighbours on each side, but Lines 50-52 clamp the first and last samples to shorter windows after Line 42 only validates the overall series length. That makes edge points easy false positives:100,100,100,100,100,100,200gets its last point flattened back to100because there is no right-hand context andwindowMadfalls to0. Skip indices that cannot form a full symmetric window, or handle boundaries explicitly.Suggested fix
Also applies to: 41-42, 48-52