Skip to content

slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes." #1883

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
3 tasks
jeffschwMSFT opened this issue Jan 24, 2024 · 7 comments
Open
3 tasks

slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes." #1883

jeffschwMSFT opened this issue Jan 24, 2024 · 7 comments
Labels
First Responder Known Build Error

Comments

@jeffschwMSFT
Copy link
Member

jeffschwMSFT commented Jan 24, 2024 •
edited by build-analysis bot
Loading

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

  • Feature changes/additions
  • Bug fixes
  • Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build Definition Step Name Console log Pull Request
2701346 dotnet-performance Performance maui_scenarios_ios FullAOT Mono osx 15 x64 iPhoneMini12 main Log
2701155 dotnet-performance Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main Log
2701106 dotnet-runtime Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0 Log
2701009 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0 Log
2700873 dotnet-runtime Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0 Log
2700836 dotnet-performance Performance maui_scenarios_ios FullAOT Mono osx 15 x64 iPhoneMini12 main Log
2700704 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0 Log
2700559 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0 Log
2700461 dotnet-performance Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main Log
2700151 dotnet-performance Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main Log
2700075 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0 Log
2700037 dotnet-performance Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main Log
2699915 dotnet-runtime Performance ios_scenarios iOSNativeAOT JIT iOSStripSymbols osx x64 perfiphone12mini net10.0 Log
2699820 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0 Log
2699796 dotnet-performance Performance maui_scenarios_ios FullAOT Mono osx 15 x64 iPhoneMini12 main Log
1031449 dotnet/performance Performance micro windows 22H2 x86 Open 9.0 Log
2698950 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0 Log
2698745 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0 Log
2698694 dotnet-performance Performance maui_scenarios_ios FullAOT Mono osx 15 x64 iPhoneMini12 main Log
2698523 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0 Log
2698356 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0 Log
2698226 dotnet-performance Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main Log
2698215 dotnet-runtime Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0 Log
1029500 dotnet/runtime osx-arm64 Debug PALTests Log
2698038 dotnet-runtime Performance ios_scenarios iOSNativeAOT JIT osx x64 perfiphone12mini net10.0 Log
2697888 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0 Log
2697822 dotnet-performance Performance maui_scenarios_ios FullAOT Mono osx 15 x64 iPhoneMini12 main Log
2697591 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0 Log
2697332 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0 Log
2697295 dotnet-performance Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main Log
2696936 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0 Log
2696918 dotnet-performance Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main Log
2696809 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0 Log
1027796 dotnet/runtime osx-x64 Debug PALTests Log
2696799 dotnet-runtime coreclr Pri1 Runtime Tests Run osx arm64 checked Log
1027776 dotnet/runtime Libraries Test Run checked coreclr osx x64 Debug Log dotnet/runtime#115081
1027773 dotnet/runtime coreclr Pri0 Runtime Tests Run osx x64 checked Log dotnet/runtime#115089
2696700 dotnet-performance Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main Log
2696517 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0 Log
2696382 dotnet-runtime Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0 Log
1027561 dotnet/performance Performance roslyn ubuntu 2204 x64 Open main Log
2696396 dotnet-performance Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main Log
2696268 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0 Log
2696167 dotnet-performance Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main Log
2696154 dotnet-runtime Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0 Log
2696034 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0 Log
2695874 dotnet-performance Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main Log
2695860 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0 Log
2695624 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0 Log
2695471 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0 Log
2695462 dotnet-performance Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main Log
2695316 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0 Log
2695127 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0 Log
2695086 dotnet-performance Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main Log
2694947 dotnet-performance Performance scenarios windows 20H1 arm64 Tiger main Log
2694852 dotnet-runtime Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0 Log
2694643 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0 Log
2694624 dotnet-performance Performance maui_scenarios_ios FullAOT Mono osx 15 x64 iPhoneMini12 main Log
2694574 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0 Log
2694455 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0 Log
2694332 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0 Log
2694213 dotnet-performance Performance maui_scenarios_ios FullAOT Mono osx 15 x64 iPhoneMini12 main Log
2694189 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0 Log
2693993 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0 Log
1024312 dotnet/arcade Linux Build_Debug Log dotnet/arcade#15770
2693841 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0 Log
2693780 dotnet-performance Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main Log
1023955 dotnet/runtime maccatalyst-arm64 Release AllSubsets_Mono Log
2693689 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0 Log
2693568 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0 Log
2693522 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0 Log
2693403 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0 Log
2693349 dotnet-performance Performance maui_scenarios_ios NativeAOT CoreCLR osx 15 x64 iPhoneMini12 main Log
2693175 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0 Log
2693147 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0 Log
2692956 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0 Log
2692871 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0 Log
2692859 dotnet-performance Performance maui_scenarios_ios FullAOT Mono osx 15 x64 iPhoneMini12 main Log
1022800 dotnet/performance Performance micro ubuntu 2204 x64 Open 9.0 Log dotnet/performance#4835
2692817 dotnet-runtime Performance ios_scenarios iOSNativeAOT JIT osx x64 perfiphone12mini net10.0 Log
1022733 dotnet/aspnetcore Test: Blazor E2E tests on Linux Log dotnet/aspnetcore#61609
2692001 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0 Log
2691752 dotnet-performance Performance roslyn ubuntu 2204 x64 Tiger main Log
1021252 dotnet/runtime coreclr Pri0 Runtime Tests Run osx x64 checked Log dotnet/runtime#114630
1021288 dotnet/runtime osx-arm64 Release CoreCLR_Release Log
1021281 dotnet/runtime osx-arm64 Release CoreCLR_Release Log
1021270 dotnet/runtime osx-x64 Release CoreCLR_Release Log
2691337 dotnet-install-scripts MacOSTests Log
2691199 dotnet-runtime coreclr Pri1 Runtime Tests Run osx arm64 checked Log
2691197 dotnet-runtime coreclr Pri1 Runtime Tests Run osx arm64 checked Log
2691098 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSStripSymbols osx x64 perfiphone12mini net10.0 Log
1021073 dotnet/performance Performance imagesharp windows 22H2 x64 Open 8.0 Log dotnet/performance#4833
2690760 dotnet-runtime Performance ios_scenarios iOSMono JIT osx x64 perfiphone12mini net10.0 Log
2690711 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0 Log
2690449 dotnet-performance Performance powershell windows 20H1 arm64 Tiger 9.0 Log
2689684 dotnet-runtime Performance ios_scenarios iOSNativeAOT JIT osx x64 perfiphone12mini net10.0 Log
2689660 dotnet-runtime Performance ios_scenarios iOSMono JIT iOSLlvmBuild iOSStripSymbols osx x64 perfiphone12mini net10.0 Log
2689130 dotnet-runtime Performance ios_scenarios iOSNativeAOT JIT osx x64 perfiphone12mini net10.0 Log
1018775 dotnet/aspnetcore Test: Blazor E2E tests on Linux Log dotnet/aspnetcore#61516
2688881 dotnet-runtime Performance ios_scenarios iOSNativeAOT JIT osx x64 perfiphone12mini net10.0 Log
Displaying 100 of 228 results

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
1 38 228
@lewing lewing mentioned this issue Feb 8, 2024
Open
3 tasks
@lewing
Copy link
Member

lewing commented Feb 9, 2024

🤔

@nagilson
Copy link
Member

nagilson commented Oct 3, 2024

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

@AaronRobinsonMSFT AaronRobinsonMSFT mentioned this issue Dec 10, 2024
Merged
@ivanpovazan
Copy link
Member

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

  • On success: Run tests in Helix step reports:
Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

  • While on failure - timeout the same step gets stuck with:
Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

@garath
Copy link
Member

garath commented Jan 27, 2025

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

@garath garath self-assigned this Jan 27, 2025
@garath
Copy link
Member

garath commented Jan 28, 2025

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.

@garath garath removed their assignment Jan 29, 2025
@ivanpovazan
Copy link
Member

ivanpovazan commented Feb 13, 2025 •
edited
Loading

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

We are still experiencing the problem.

Should we try to change the queue to osx.13.arm64?

@dotnet/dnceng

This was referenced Feb 18, 2025
Open
Closed
@ilyas1974
Copy link
Contributor

Looks at the failing builds, I'm noticing the jobs that are timing out in the hosted pool (Azure Pipelines) are still using the older hardware. I would recommend moving the workloads to the mac-latest-internal or mac-14-arm64 agent specifications. These have the latest mac hardware associated with them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
First Responder Known Build Error
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@lewing @garath @jeffschwMSFT @ilyas1974 @nagilson @ivanpovazan