slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1314835	dotnet/aspnetcore	Tests: macOS	Log
1314804	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log
1314797	dotnet/aspnetcore	Tests: macOS	Log
2915165	dotnet-runtime	Performance ios_scenarios iOSCoreCLR Interpreter iOSStripSymbols osx x64 perfiphone12mini net11.0	Log
1314746	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#65574
2915092	dotnet-runtime	Performance ios_scenarios iOSMono FullAOT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net11.0	Log
1314457	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#65567
2914881	dotnet-maui	Controls (v18.5) Cells,CheckBox,ContextActions,CustomRenderers,DatePicker,Dispatcher,DisplayAlert,DisplayPrompt,DragAndDrop
1314423	dotnet/aspnetcore	Tests: macOS	Log
2914871	dotnet-maui	Controls CollectionView	Log
2914866	dotnet-runtime	Libraries Test Run checked coreclr osx arm64 Release	Log
2914841	dotnet-runtime	Performance ios_scenarios iOSNativeAOT NativeAOT iOSStripSymbols osx x64 perfiphone12mini net11.0	Log
2914792	dotnet-runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
1314254	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#65562
2914706	dotnet-runtime	Performance ios_scenarios iOSNativeAOT NativeAOT iOSStripSymbols osx x64 perfiphone12mini net11.0	Log
1314191	dotnet/aspnetcore	Tests: macOS	Log
1314142	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log
2914361	dotnet-maui	Controls (vlatest) CollectionView	Log
1313535	dotnet/maui	Controls CollectionView	Log
1313785	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#65558
1313431	dotnet/maui	Controls CollectionView	Log
2914501	dotnet-runtime	Performance ios_scenarios iOSMono FullAOT iOSStripSymbols osx x64 perfiphone12mini net11.0	Log
2914352	dotnet-maui	Controls (v18.4) SoftInput,Stepper,Switch,SwipeView	Log
2914106	dotnet-runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	#58446
1312939	dotnet/maui	Controls CollectionView	Log
2914301	dotnet-maui	Controls CollectionView	Log
2914175	dotnet-maui	Controls CollectionView	Log
1313080	dotnet/maui	Controls Layout	Log	dotnet/maui#34275
1313074	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
2914245	dotnet-runtime	Performance ios_scenarios iOSMono FullAOT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net11.0	Log
1312991	dotnet/maui	Controls Shell	Log	dotnet/maui#34237
1313005	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
2914067	dotnet-sdk	TestBuild: macOS (x64)	Log	#58430
2914147	dotnet-runtime	Performance ios_scenarios iOSCoreCLR R2R iOSStripSymbols osx x64 perfiphone12mini net11.0	Log
2914127	dotnet-runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	#58448
2914104	dotnet-runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	#57892
1310719	dotnet/runtime	Libraries Test Run checked coreclr osx x64 Debug	Log	dotnet/runtime#124885
1312741	dotnet/sdk	TestBuild: macOS (x64)	Log	dotnet/sdk#53185
1312735	dotnet/sdk	TestBuild: macOS (arm64)	Log	dotnet/sdk#53184
1312602	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#123138
1312635	dotnet/runtime	tvos-arm64 Release AllSubsets_CoreCLR_Smoke	Log	dotnet/runtime#124927
1310803	dotnet/xharness	Helix Tests Build_Debug	Log	dotnet/xharness#1542
1310808	dotnet/xharness	Helix Tests Build_Debug	Log	dotnet/xharness#1543
1312556	dotnet/aspnetcore	Tests: macOS	Log
2913882	dotnet-runtime	Performance ios_scenarios iOSMono FullAOT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net11.0	Log
1312336	dotnet/runtime	tvossimulator-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log
2913823	dotnet-efcore	macOS	Log
1312205	dotnet/aspnetcore	Tests: macOS	Log
2913603	dotnet-maui	Controls (vlatest) Entry	Log
1311033	dotnet/maui	Controls Layout	Log	dotnet/maui#33687
2913671	dotnet-runtime	Performance ios_scenarios iOSCoreCLR Interpreter iOSStripSymbols osx x64 perfiphone12mini net11.0	Log
1311765	dotnet/maui	Controls Layout	Log	dotnet/maui#32878
1311815	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
1311737	dotnet/maui	Controls Shell	Log
1311581	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log
2913452	dotnet-maui	Controls (v18.4)
2913528	dotnet-maui	Controls (v18.4) Label	Log
2913406	dotnet-maui	Controls (v18.4) Cells,CheckBox,ContextActions,CustomRenderers,DatePicker,Dispatcher,DisplayAlert,DisplayPrompt,DragAndDrop
1311426	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#123611
1311326	dotnet/runtime	osx-x64 release Runtime_Release	Log	dotnet/runtime#124854
1311324	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR	Log	dotnet/runtime#124854
1311270	dotnet/xharness	Helix Tests Build_Debug	Log	dotnet/xharness#1544
2913439	dotnet-runtime	Performance ios_scenarios iOSNativeAOT NativeAOT iOSStripSymbols osx x64 perfiphone12mini net11.0	Log
2913270	dotnet-maui	Controls (vlatest) Shell	Log
1311219	dotnet/runtime	ios-arm64 Release AllSubsets_Mono_Smoke	Log	dotnet/runtime#124902
1311147	dotnet/maui	Controls Layout	Log
1311078	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#64964
1311050	dotnet/maui	Controls Layout	Log	dotnet/maui#34240
1310815	dotnet/maui	Controls CollectionView	Log
1310998	dotnet/runtime	ios-arm64 Release AllSubsets_Mono_Smoke	Log	dotnet/runtime#123145
2913331	dotnet-performance	Performance nativeaot_scenarios windows 22H2 arm64 Ampere main	Log
1310894	dotnet/runtime	osx-arm64 Debug Libraries_CheckedCoreCLR		dotnet/runtime#123860
1310877	dotnet/maui	Controls TabbedPage,TableView,TimePicker,TitleView,ToolbarItem	Log	dotnet/maui#33982
1310880	dotnet/maui	Controls TabbedPage,TableView,TimePicker,TitleView,ToolbarItem	Log	dotnet/maui#33985
2913171	dotnet-maui	Controls (v18.4) Editor,Effects,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView
2913255	dotnet-maui	Controls CollectionView	Log
1310643	dotnet/maui	Controls Navigation	Log	dotnet/maui#30878
1310547	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_CoreCLR_RuntimeTests	Log
2912808	dotnet-maui	Controls (v18.4) ScrollView
1310389	dotnet/aspnetcore	Tests: macOS	Log
2912897	dotnet-runtime	Performance ios_scenarios iOSMono FullAOT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net11.0	Log
2912544	dotnet-maui	Controls (v18.4) TabbedPage,TableView,TimePicker,TitleView,ToolbarItem	Log
1310078	dotnet/maui	Controls Layout	Log	dotnet/maui#34203
2912721	dotnet-maui	Controls (v18.4) Layout
1309951	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_RuntimeTests	Log
1309942	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#65460
1309673	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#124817
2912510	dotnet-maui	Controls (v18.4) Navigation
2912513	dotnet-maui	Controls (v18.4) Layout	Log
1309303	dotnet/maui	Controls RadioButton,RefreshView	Log
2912268	dotnet-maui	Controls (v18.5) RadioButton,RefreshView	Log
1309353	dotnet/runtime	osx-arm64 Debug CoreCLR_Libraries	Log	dotnet/runtime#124833
1309268	dotnet/maui	Controls Layout	Log
1309212	dotnet/aspnetcore	Tests: macOS	Log
2912411	dotnet-runtime	Performance ios_scenarios iOSCoreCLR Interpreter iOSStripSymbols osx x64 perfiphone12mini net11.0	Log
1309071	dotnet/aspnetcore	Tests: macOS	Log	dotnet/aspnetcore#65460
2911892	dotnet-maui	Controls (vlatest) ViewBaseTests,Window	Log
2912293	dotnet-maui	Controls Navigation	Log
1308956	dotnet/maui	Controls Layout		dotnet/maui#34121
2912185	dotnet-runtime	Performance ios_scenarios iOSNativeAOT NativeAOT iOSStripSymbols osx x64 perfiphone12mini net11.0	Log
Displaying 100 of 851 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
8	190	830

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

[ivanpovazan]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

[ilyas1974]

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.