slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1400007	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127510
1399536	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127058
1399975	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#54115
2962230	dotnet-maui	RunOniOS_BlazorRelease ARM64	Log
1399661	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#54122
1399821	dotnet/maui	Controls Shell	Log	dotnet/maui#35072
2962160	dotnet-maui	RunOniOS_MauiRelease x64	Log
1399843	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#125452
1399801	dotnet/maui	Controls ScrollView	Log	dotnet/maui#34923
1399727	dotnet/runtime	osx-x64 Debug Installer_Build_And_Test	Log	dotnet/runtime#126701
1399815	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit	Log	dotnet/runtime#127485
2962164	dotnet-maui	RunOniOS_BlazorDebug_CoreCLR ARM64	Log
2962158	dotnet-maui	RunOniOS_BlazorRelease x64
2962165	dotnet-maui	Controls ScrollView	Log
1399628	dotnet/runtime	osx-x64 Debug Installer_Build_And_Test	Log	dotnet/runtime#127438
1399599	dotnet/runtime	osx-x64 Debug Installer_Build_And_Test	Log	dotnet/runtime#127498
1399748	dotnet/maui	Controls Layout	Log	dotnet/maui#34573
1399774	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#54101
2962154	dotnet-maui	RunOniOS_MauiReleaseTrimFull ARM64	Log
1399689	dotnet/maui	Controls Shell	Log
2962161	dotnet-maui	Controls CollectionView	Log
1399731	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#126360
1399405	dotnet/maui	Controls CollectionView	Log	dotnet/maui#35151
1399750	dotnet/sdk	TestBuild: macOS (x64)	Log
1399374	dotnet/maui	Controls Layout	Log	dotnet/maui#32674
1399683	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#54127
1399680	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#54126
1399665	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#54124
2962156	dotnet-maui	Build Device Tests (Mono)	Log
1399619	dotnet/runtime	osx-x64 Release AllSubsets_Mono_Interpreter_RuntimeTests monointerpreter	Log	dotnet/runtime#127238
1399622	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#127148
1399641	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#54120
1399640	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#54119
1399604	dotnet/runtime	osx-x64 Release Libraries_Release_CoreCLR	Log	dotnet/runtime#127498
1399596	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127497
1399541	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127464
2962054	dotnet-efcore	macOS	Log
2962049	dotnet-roslyn	Build_macOS_Debug	Log
1399432	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126277
1399399	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#124738
1399392	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126351
1384827	dotnet/maui	Controls Layout	Log	dotnet/maui#34980
1399358	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126932
1399355	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127493
1399351	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#126367
2961925	dotnet-roslyn	Build_macOS_Debug	Log
1398788	dotnet/maui	Controls Layout	Log	dotnet/maui#34737
1399108	dotnet/runtime	tvossimulator-arm64 Debug AllSubsets_Mono	Log	dotnet/runtime#126367
1399089	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#127485
2961625	dotnet-maui	Controls Shell	Log
1399032	dotnet/aspnetcore	Tests: macOS	Log
1398572	dotnet/maui	Controls CollectionView	Log	dotnet/maui#34614
1398514	dotnet/maui	Controls CollectionView	Log
1398520	dotnet/maui	Controls Layout	Log	dotnet/maui#35086
1398502	dotnet/maui	Controls Layout	Log
1398840	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127242
2961548	dotnet-maui	Controls Shell	Log
1398755	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#126765
1398717	dotnet/runtime	ios-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#126236
1398791	dotnet/aspnetcore	Tests: macOS	Log
1398689	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log	dotnet/runtime#127465
1398685	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#126569
1398665	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127088
1398657	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127470
1398294	dotnet/maui	Controls Shell	Log	dotnet/maui#34758
2961584	dotnet-maui	RunOniOS_MauiReleaseTrimFull x64	Log
1398570	dotnet/maui	Controls Layout	Log	dotnet/maui#31672
2961547	dotnet-maui	RunOniOS_MauiRelease x64	Log
2961541	dotnet-maui	RunOniOS_MauiRelease ARM64	Log
2961370	dotnet-maui	Controls Shell	Log
1397840	dotnet/maui	Controls Shell	Log
1398504	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127225
1398506	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#127225
1398542	dotnet/roslyn	Build_macOS_Debug	Log
1398469	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#126980
1398140	dotnet/maui	Controls CollectionView	Log
1398416	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127195
1398392	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127292
1398400	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#127134
2961187	dotnet-maui	Controls (vlatest) WebView	Log
1398355	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66443
1398093	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#125024
2961196	dotnet-maui	RunOniOS_MauiReleaseTrimFull ARM64	Log
2961199	dotnet-maui	RunOniOS_MauiRelease ARM64	Log
1397972	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#53947
2961186	dotnet-maui	RunOniOS_MauiReleaseTrimFull ARM64	Log
1397946	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127451
2961175	dotnet-maui	RunOniOS_MauiNativeAOT ARM64	Log
1397893	dotnet/maui	Controls Layout	Log
2961233	dotnet-performance	Performance mlnet ubuntu 2204 arm64 Ampere 8.0	Log
1397906	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127058
2961176	dotnet-maui	Controls Shell	Log
2961159	dotnet-maui	Build macOS	Log
1397846	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#66489
1397789	dotnet/maui	Controls Shell	Log	dotnet/maui#29591
1395687	dotnet/maui	Controls CollectionView	Log	dotnet/maui#35091
1397616	dotnet/maui	Controls Shell	Log	dotnet/maui#35131
1397780	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#127383
1397613	dotnet/maui	Controls CollectionView	Log	dotnet/maui#35008
1397708	dotnet/runtime	coreclr Pri1 Runtime Tests Run osx x64 checked	Log
Displaying 100 of 1368 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
81	288	1368

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.