[Storage] Change default parallelism to CPU-based value by tanyasethi-msft · Pull Request #26880 · Azure/azure-sdk-for-go

tanyasethi-msft · 2026-05-25T12:26:26Z

Summary

Changes default concurrency for parallel uploads/downloads from fixed values (5 for buffer/file, 1 for stream) to Clamp(NumCPU, 8, 96)
Adds DefaultConcurrencyValue() function (keeps old DefaultConcurrency const deprecated for backward compat)
Adds AZURE_STORAGE_USE_LEGACY_DEFAULT_CONCURRENCY=true env var opt-out to revert to old defaults without code changes

Performance Impact

Benchmarked on Standard_D32s_v6 against a standard storage account, 4 MiB chunk size, 3 iterations per size.

UploadBuffer — OLD (concurrency=5) vs NEW (concurrency=32)

Size	OLD (5)	NEW (32)	Improvement
256 MB	94 MB/s	90 MB/s	~same
512 MB	165 MB/s	584 MB/s	+254%
1 GiB	176 MB/s	742 MB/s	+322%
2 GiB	177 MB/s	897 MB/s	+407%
4 GiB	168 MB/s	1,008 MB/s	+500%

UploadStream — OLD (concurrency=1) vs NEW (concurrency=32)

Size	OLD (1)	NEW (32)	Improvement
256 MB	87 MB/s	287 MB/s	+230%
512 MB	88 MB/s	375 MB/s	+326%
1 GiB	90 MB/s	483 MB/s	+437%
2 GiB	89 MB/s	503 MB/s	+465%
4 GiB	86 MB/s	542 MB/s	+530%

DownloadBuffer — OLD (concurrency=5) vs NEW (concurrency=32)

Size	OLD (5)	NEW (32)	Improvement
256 MB	205 MB/s	251 MB/s	+22%
512 MB	168 MB/s	459 MB/s	+173%
1 GiB	150 MB/s	654 MB/s	+336%
2 GiB	155 MB/s	756 MB/s	+388%
4 GiB	151 MB/s	822 MB/s	+444%

Key Observations

UploadStream sees the largest gains — it was previously sequential (concurrency=1), now fully parallel
UploadBuffer at 4 GiB: 6x throughput improvement (168 → 1,008 MB/s), saturating the VM's ~12.5 Gbps network
DownloadBuffer at 4 GiB: 5.4x throughput improvement (151 → 822 MB/s)
Small blobs (<256 MB) see minimal change — single-PUT path is unaffected
Numbers scale with VM size; larger VMs with higher network bandwidth (e.g., 100 Gbps) will see proportionally higher absolute throughput

Opt-out

Customers who experience issues can revert to the old default without code changes:

export AZURE_STORAGE_USE_LEGACY_DEFAULT_CONCURRENCY=true

…, azfile, azdatalake Update default concurrency for uploads and downloads from a fixed value of 5 (or 1 for stream uploads) to Clamp(NumCPU, 8, 96), matching the Rust SDK implementation. This improves throughput on modern multi-core VMs while clamping prevents resource exhaustion on small or very large machines. Adds AZURE_STORAGE_USE_LEGACY_DEFAULT_CONCURRENCY=true env var opt-out to revert to the previous default of 5.

…tream

Copilot

Pull request overview

This PR updates storage transfer defaults so azblob, azfile, and azdatalake choose parallelism from CPU count instead of fixed legacy values, with an environment-variable opt-out for compatibility.

Changes:

Adds DefaultConcurrencyValue() in shared storage helpers and applies it to batch transfers and stream uploads.
Updates option comments to describe CPU-based concurrency defaults.
Adds unit coverage for default concurrency selection and live benchmarks for azblob parallelism.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
`sdk/storage/azfile/internal/shared/batch_transfer.go`	Adds CPU-based default concurrency helper and uses it for batch transfers.
`sdk/storage/azfile/internal/shared/batch_transfer_test.go`	Adds tests for concurrency bounds, CPU matching, determinism, and legacy env var.
`sdk/storage/azfile/file/models.go`	Updates file transfer option docs and stream upload defaulting.
`sdk/storage/azdatalake/internal/shared/batch_transfer.go`	Adds CPU-based default concurrency helper and uses it for batch transfers.
`sdk/storage/azdatalake/internal/shared/batch_transfer_test.go`	Adds tests for concurrency bounds, CPU matching, determinism, and legacy env var.
`sdk/storage/azdatalake/file/models.go`	Updates datalake transfer option docs and stream upload defaulting.
`sdk/storage/azblob/internal/shared/shared_test.go`	Adds tests for shared concurrency default behavior.
`sdk/storage/azblob/internal/shared/batch_transfer.go`	Adds CPU-based default concurrency helper and uses it for batch transfers.
`sdk/storage/azblob/blockblob/parallelism_bench_test.go`	Adds live benchmarks for upload/download parallelism behavior.
`sdk/storage/azblob/blockblob/models.go`	Updates block blob transfer option docs and stream upload defaulting.
`sdk/storage/azblob/blockblob/chunkwriting_test.go`	Adjusts buffer allocation assertion for higher default stream concurrency.
`sdk/storage/azblob/blob/models.go`	Updates blob download option docs for CPU-based defaults.
`sdk/storage/azblob/blob/constants.go`	Deprecates fixed `DefaultConcurrency` and exposes `DefaultConcurrencyValue()`.

…wn paragraph

souravgupta-msft · 2026-05-26T07:03:49Z

 func (u *UploadStreamOptions) setDefaults() {
 	if u.Concurrency == 0 {
-		u.Concurrency = 1
+		u.Concurrency = int(shared.DefaultConcurrencyValue())


This method returns 5 if AZURE_STORAGE_USE_LEGACY_DEFAULT_CONCURRENCY environment variable is set. Whereas the previous default value for concurrency in UploadStreamOptions is 1.

souravgupta-msft · 2026-05-26T07:05:59Z

 func (u *UploadStreamOptions) setDefaults() {
 	if u.Concurrency == 0 {
-		u.Concurrency = 1
+		u.Concurrency = shared.DefaultConcurrencyValue()


Same as previous, this returns the previous default value as 5 instead of 1

souravgupta-msft · 2026-05-26T07:08:52Z

 func (u *UploadStreamOptions) setDefaults() {
 	if u.Concurrency == 0 {
-		u.Concurrency = 1
+		u.Concurrency = int(shared.DefaultConcurrencyValue())


Same as previous

souravgupta-msft · 2026-05-26T07:13:51Z

+	// DefaultConcurrency is the legacy default number of blocks downloaded or uploaded in parallel.
+	//
+	// Deprecated: Use DefaultConcurrencyValue() instead, which returns a value based on CPU core count.
+	DefaultConcurrency = shared.DefaultConcurrency //nolint:staticcheck // intentional re-export of deprecated const for backward compat


Add changelog entries in all the three packages

souravgupta-msft · 2026-05-26T07:15:23Z

+	// Each concurrent upload will create a buffer of size BlockSize.  The default is based on
+	// CPU core count (min 8, max 96). Set AZURE_STORAGE_USE_LEGACY_DEFAULT_CONCURRENCY=true to revert to the previous default.
 	Concurrency int


Please check if data validation tests are already there for all the affected upload and download methods.

tanyasethi-msft added 2 commits May 25, 2026 16:29

Add GiB-scale parallelism benchmarks for upload/download buffer and s…

43cd16f

…tream

Copilot AI review requested due to automatic review settings May 25, 2026 12:26

tanyasethi-msft requested review from gapra-msft, jhendrixMSFT and souravgupta-msft as code owners May 25, 2026 12:26

Copilot started reviewing on behalf of tanyasethi-msft May 25, 2026 12:26 View session

github-actions Bot added the Storage Storage Service (Queues, Blobs, Files) label May 25, 2026

Copilot AI reviewed May 25, 2026

View reviewed changes

tanyasethi-msft added 5 commits May 25, 2026 18:04

Remove duplicate test files from azfile and azdatalake

f35f22e

Remove benchmark file from PR

0ac9073

Address review comments: hermetic tests and env var in doc comments

6256896

Fix gocritic deprecatedComment: separate Deprecated notice into its o…

236bfd8

…wn paragraph

Fix staticcheck SA1019 for deprecated const re-export

18eb0ba

souravgupta-msft reviewed May 26, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Storage] Change default parallelism to CPU-based value#26880

[Storage] Change default parallelism to CPU-based value#26880
tanyasethi-msft wants to merge 7 commits into
mainfrom
feat/storage/parallelism

tanyasethi-msft commented May 25, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

souravgupta-msft May 26, 2026

Uh oh!

souravgupta-msft May 26, 2026

Uh oh!

souravgupta-msft May 26, 2026

Uh oh!

souravgupta-msft May 26, 2026

Uh oh!

souravgupta-msft May 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

tanyasethi-msft commented May 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Performance Impact

UploadBuffer — OLD (concurrency=5) vs NEW (concurrency=32)

UploadStream — OLD (concurrency=1) vs NEW (concurrency=32)

DownloadBuffer — OLD (concurrency=5) vs NEW (concurrency=32)

Key Observations

Opt-out

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

souravgupta-msft May 26, 2026

Choose a reason for hiding this comment

Uh oh!

souravgupta-msft May 26, 2026

Choose a reason for hiding this comment

Uh oh!

souravgupta-msft May 26, 2026

Choose a reason for hiding this comment

Uh oh!

souravgupta-msft May 26, 2026

Choose a reason for hiding this comment

Uh oh!

souravgupta-msft May 26, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tanyasethi-msft commented May 25, 2026 •

edited

Loading