Skip to content

Shade by avara submission#62

Open
kevin-romer wants to merge 2 commits into
thedatumorg:TSB-AD-algofrom
kevin-romer:shade-by-avara-submission
Open

Shade by avara submission#62
kevin-romer wants to merge 2 commits into
thedatumorg:TSB-AD-algofrom
kevin-romer:shade-by-avara-submission

Conversation

@kevin-romer
Copy link
Copy Markdown

Summary

  • Add benchmark_exp/Run_SHADE_by_Avara.py — a new pretrained anomaly detector submission based on SHADE (Sparse Hybrid Anomaly Detection Engine), a hosted time-series anomaly detection endpoint
  • Update uni_mergedTable_VUS-PR.csv and multi_mergedTable_VUS-PR.csv with SHADE evaluation results
  • SHADE achieves new state-of-the-art VUS-PR results on both TSB-AD-U and TSB-AD-M

Results

TSB-AD-U (Univariate)

Rank Model VUS-PR Pretrained
1 SHADE 0.59 Yes
2 TSPulse (FT) 0.55 Yes
3 Time-RCD 0.52 Yes

TSB-AD-M (Multivariate)

Rank Model VUS-PR Pretrained
1 SHADE 0.50 Yes
2 xLSTMAD 0.39 No
2 TSPulse (FT) 0.39 Yes

Running Instructions

pip install TSB-AD

Set endpoint credentials:

export SHADE_API_KEY='<provided privately>'
export SHADE_BASE_URL='https://shade.avara-ai.com'

Run the detector through the TSB-AD benchmark runner:

python benchmark_exp/Run_SHADE_by_Avara.py \
  --file_lsit Datasets/File_List/TSB-AD-U-Eva.csv \
  --data_direc /path/to/TSB-AD-U \
  --AD_Name SHADE_Lambda_U \
  --score_dir eval/score/uni \
  --save_dir eval/metrics/uni \
  --save true \
  --overwrite true

For multivariate evaluation:

python benchmark_exp/Run_SHADE_by_Avara.py \
  --file_lsit Datasets/File_List/TSB-AD-M-Eva.csv \
  --data_direc /path/to/TSB-AD-M \
  --AD_Name SHADE_Lambda_M \
  --score_dir eval/score/multi \
  --save_dir eval/metrics/multi \
  --save true \
  --overwrite true

Note: Run_SHADE_by_Avara.py connects to a hosted SHADE endpoint. Please reach out privately for the SHADE_API_KEY required for evaluation. The script sends only the input values, sanitized series name, split name, and train-prefix length to the endpoint. Labels are not sent to the endpoint and are used only locally by the TSB-AD evaluation code.

Files Changed

  • benchmark_exp/Run_SHADE_by_Avara.py
  • uni_mergedTable_VUS-PR.csv
  • multi_mergedTable_VUS-PR.csv

Adds `Run_SHADE_by_Avara.py` as a SHADE community submission for TSB-AD.
Update U merged benchmark results with recomputed VUS-PR values

Recompute and fill the 62 SHADE_Lambda_U VUS-PR rows that were blank in the
endpoint metrics file using saved score arrays, labels, and the official TSB-AD
VUS-PR curve code.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant