[ML] Reapply: Run allowlist validation in PyTorch edge pipeline#3007
Merged
edsavage merged 9 commits intoelastic:mainfrom Mar 26, 2026
Merged
[ML] Reapply: Run allowlist validation in PyTorch edge pipeline#3007edsavage merged 9 commits intoelastic:mainfrom
edsavage merged 9 commits intoelastic:mainfrom
Conversation
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
edsavage
force-pushed
the
revert/pr-3005
branch
3 times, most recently
from
592e78c to
e0b3d61
Compare
Contributor
Author
|
buildkite run_pytorch_tests |
Contributor
Author
|
buildkite run_pytorch_tests |
4 similar comments
Contributor
Author
|
buildkite run_pytorch_tests |
Contributor
Author
|
buildkite run_pytorch_tests |
Contributor
Author
|
buildkite run_pytorch_tests |
Contributor
Author
|
buildkite run_pytorch_tests |
…tic#2989)" (elastic#3005) This reverts commit 9cc49ff.
The Linux build/test Docker images don't include Python 3 (it's only used during image builds to compile PyTorch, then dropped in the multi-stage final image). Move the validation to a dedicated pipeline step using a python:3 agent image, triggered only for run_pytorch_tests builds. Made-with: Cursor
The python:3 tag now resolves to Python 3.14, which doesn't have torch==2.7.1 wheels. Pin to python:3.12 to match the PyTorch version we build and ship against. Made-with: Cursor
The step was being killed (exit -1) with no output — likely OOM or disk exhaustion from installing torch (800MB+) and tracing 27+ models. Add memory (16G), ephemeral storage (20G), and a 60-minute timeout. Remove -q from pip install so progress is visible in logs. Made-with: Cursor
The validation step should fail the build if it detects allowlist errors — that's the whole point of running it. The upload step retains soft_fail in case of pipeline upload issues. Made-with: Cursor
Without a notify/github_commit_status block, the step doesn't appear as a check on the GitHub PR. Add it so the validation result is visible alongside the other build/test checks. Made-with: Cursor
Private Elastic models on HuggingFace (elastic/elser-v2, etc.) can't be downloaded without a HF_TOKEN, causing the validation step to fail in CI even though the ops are correct. Change validate_model() to return "pass"/"fail"/"skip" — load/trace failures are reported as skips (warnings) while op validation failures remain hard failures. Also pass auto_class and config_overrides through to support BART and QA models. Made-with: Cursor
edsavage
force-pushed
the
revert/pr-3005
branch
from
ef27199 to
e1ff7da
Compare
Contributor
Author
|
buildkite run_pytorch_tests |
facebook/bart-large-mnli is 1.63GB — loading it into memory for tracing after torch and 30 other models exhausted the 16GB limit. Bump to 32GB memory and 30GB ephemeral storage. Made-with: Cursor
When validating 30+ models sequentially, the HF model weights accumulate in memory. Explicitly delete the original model, tokenizer, and inputs after tracing, and gc.collect() after each validation to release memory promptly. This should allow the validation step to complete within 32GB for the full model set including facebook/bart-large-mnli (1.63GB). Made-with: Cursor
Contributor
Author
|
buildkite run_pytorch_tests |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Made with Cursor