buildkit: deploy on meta-prod-aws-ue1 (us-east-1)#728
Merged
Conversation
[ghstack-poisoned]
This was referenced Jun 10, 2026
tofu plan — meta-prod-aws-ue1✅ Plan succeeded · commit Plan output |
[ghstack-poisoned]
huydhn
added a commit
that referenced
this pull request
Jun 10, 2026
Same buildkit config as arc-cbr-production (ue2): m6id.24xlarge amd64 (2/node) + m7gd.16xlarge arm64 (4/node), autoscaling min 2/4, max 360/30, fallback 32/8. Adds keda + buildkit to the cluster's module list (keda provides the CRDs the autoscaling needs). ghstack-source-id: f6bf677 Pull-Request: #728
tofu plan — arc-cbr-production✅ Plan succeeded · commit Plan output |
Contributor
Author
|
I suspect that there might be more to this than meets the eye here, not the autoscaling part, but the Buildkit in multiple regions part. @jeanschmidt @zxiiro If you have successfully deployed Buildkit in multiple regions, let me know. I'm referring to these lines specifically https://github.com/pytorch/ci-infra/blob/main/osdc/modules/arc-runners/defs/rel-l-x86iavx512-8-64.yaml#L11-L12 |
jeanschmidt
approved these changes
Jun 11, 2026
[ghstack-poisoned]
huydhn
added a commit
that referenced
this pull request
Jun 11, 2026
Same buildkit config as arc-cbr-production (ue2): m6id.24xlarge amd64 (2/node) + m7gd.16xlarge arm64 (4/node), autoscaling min 2/4, max 360/30, fallback 32/8. Adds keda + buildkit to the cluster's module list (keda provides the CRDs the autoscaling needs). ghstack-source-id: 0170c40 Pull-Request: #728
tofu plan — arc-cbr-production-uw1✅ Plan succeeded · commit Plan output |
[ghstack-poisoned]
huydhn
added a commit
to huydhn/pytorch-ci-infra
that referenced
this pull request
Jun 12, 2026
Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.15.0) (oldest at bottom): * pytorch#727 * pytorch#728 * pytorch#726 * pytorch#725 * __->__ pytorch#724 Same min per arch as staging (amd64 2 / arm64 4). Max sized from 14-day docker-build concurrency: amd64 128 (peak 105 + headroom), arm64 16 (peak 8, likely capped by the old fixed pool).
[ghstack-poisoned]
huydhn
added a commit
to huydhn/pytorch-ci-infra
that referenced
this pull request
Jun 12, 2026
Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.15.0) (oldest at bottom): * pytorch#727 * pytorch#728 * pytorch#726 * __->__ pytorch#725 Burst 8 parallel buildctl builds per arch (each holds a maxconn=1 slot ~10m via sleep). With amd64_min=2 they serialize ~43m > timeout 30m and fail unless KEDA scales the pool up; one wave ~18m when it does.
huydhn
added a commit
to huydhn/pytorch-ci-infra
that referenced
this pull request
Jun 12, 2026
Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.15.0) (oldest at bottom): * pytorch#727 * pytorch#728 * __->__ pytorch#726 * pytorch#725 Enable the KEDA operator's Prometheus endpoint (prometheus.operator.enabled) and add a ServiceMonitor scraping keda_* (scaler/scaledobject values, errors, activity, latency) at 60s. Lets us see what KEDA reads and when it errors / falls back.
Contributor
Author
|
This is ready to land now given the multiple Buildkit clusters test on staging #667, but I think I will hold on to this PR a bit longer until after the next prod deployment |
zxiiro
approved these changes
Jun 18, 2026
[ghstack-poisoned]
tofu plan — lf-prod-aws-ue1✅ Plan succeeded · commit Plan output |
tofu plan — lf-prod-aws-ue2✅ Plan succeeded · commit Plan output |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack from ghstack (oldest at bottom):
Same buildkit config as arc-cbr-production (ue2): m6id.24xlarge amd64 (2/node)
Adds keda + buildkit to the cluster's module list (keda provides the CRDs the
autoscaling needs).