restarted. Azure DevOps can't recover from restarts. #3879

lewing · 2024-08-26T19:12:06Z

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=784852

Build leg reported

Build / linux-x64 debug Libraries_AllConfigurations

Pull Request

dotnet/runtime#106599

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "restarted. Azure DevOps can't recover from restarts.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

Feature changes/additions
Bug fixes
Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=784852
Error message validated: [restarted. Azure DevOps can't recover from restarts.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 8/26/2024 7:12:18 PM UTC

Report

Build	Definition	Step Name	Pull Request
848347	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#108876
848348	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations	dotnet/runtime#108876
848327	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading
848321	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#108876
848282	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#109053
848262	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#102144
848200	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#108876
2564576	dotnet-dotnet	AzureLinux_x64_Cross_Alpine_arm
848006	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#109028
848007	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations	dotnet/runtime#109028
847999	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#108876
847977	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading
847967	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#108806
847801	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#108876
847802	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations	dotnet/runtime#108876
847712	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#109028
847696	dotnet/runtime	android-arm64 Release AllSubsets_Mono
847660	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#108464
847508	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#106777
847651	dotnet/sdk	LinuxBionic_NativeAOT_Shortstack_x64	dotnet/sdk#44306
847633	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations	dotnet/runtime#108939
847574	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#107166
846075	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#108352
847491	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#108876
847485	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading
2564021	dotnet-dotnet	AzureLinux_x64_Cross_ShortStack_Mono_LLVMAOT_arm64
2563840	dotnet-dotnet	AzureLinux_x64_Cross_Alpine_x64
847170	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading
847017	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#107769
846952	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#108806
846739	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#108968
846570	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#108636
846537	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#107889
846532	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#108982
846401	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#108482
846366	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#108636
846301	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#108701
846206	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#108324
846183	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#108188
845938	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading
845879	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#108701
845885	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#108757
845438	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#108482
845530	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#107843
2562362	dotnet-dotnet	Ubuntu2404_x64
2562262	dotnet-sdk	PGO: linux (x64)
845009	dotnet/runtime	browser-wasm linux Release LibraryTests
844992	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#108620
844862	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#108685
844629	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#108925
844468	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#108701
844331	dotnet/sdk	TestAsTools: windows (x64)	dotnet/sdk#44152
844111	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#107889
844098	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#108910
843842	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#107843
843776	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading
843503	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations	dotnet/runtime#108853
843221	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#108701
843118	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#108837
843043	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#108670
842876	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#108876
842846	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#108757
2560687	dotnet-dotnet	AzureLinux_x64_Cross_x64
842587	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations	dotnet/runtime#108867
842416	dotnet/runtime	CoreCLR Pri1 Runtime Tests Run windows x64 checked
842558	dotnet/sdk	AoT: windows (x64)	dotnet/sdk#43889
842319	dotnet/runtime	CoreCLR Pri0 Runtime Tests Run windows arm64 checked	dotnet/runtime#108300
2560451	dotnet-roslyn	Build_Windows_Debug
842277	dotnet/runtime	browser-wasm linux Release LibraryTests
842306	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations	dotnet/runtime#107843
842057	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#108748
841320	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#108701
841950	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#108837
841904	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#107909
841753	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#104608
841616	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#108748
841312	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#108819
2559755	dotnet-dotnet	LinuxBionic_Shortstack_x64
841284	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#108757
841176	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations	dotnet/runtime#108818
841154	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations	dotnet/runtime#108770
2559477	dotnet-dotnet	AzureLinux_x64_Cross_x64
841104	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations	dotnet/runtime#108636
841089	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations	dotnet/runtime#108707
841081	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations	dotnet/runtime#108757
841039	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations	dotnet/runtime#102249
841038	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#102249
841022	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations	dotnet/runtime#108819
841004	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations	dotnet/runtime#103375
840993	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations	dotnet/runtime#108819
840978	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations	dotnet/runtime#108818
840934	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations	dotnet/runtime#108808
840921	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations	dotnet/runtime#103375
840878	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations	dotnet/runtime#103375
840767	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations	dotnet/runtime#103375
840766	dotnet/runtime	browser-wasm linux Release LibraryTests	dotnet/runtime#103375
840745	dotnet/runtime	browser-wasm linux Release LibraryTests_Threading	dotnet/runtime#108757
840746	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations	dotnet/runtime#108757
840674	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations	dotnet/runtime#103375
840592	dotnet/runtime	linux-x64 debug Libraries_AllConfigurations	dotnet/runtime#108808
Displaying 100 of 682 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
6	79	682

The text was updated successfully, but these errors were encountered:

am11 · 2024-10-07T05:54:23Z

It seems like all reports are pointing to linux-x64 dev-innerloop leg from this definition: https://dev.azure.com/dnceng-public/public/_build?definitionId=133. GitHub doesn't sync the status and keeps showing as if the job is running for days.. Opened dotnet/runtime#108581 to disable the leg.

am11 · 2024-10-07T12:57:20Z

Just before the timeout we see low memory warnings like these:

https://dev.azure.com/dnceng-public/public/_build/results?buildId=829130&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857

then it hangs for ~20 minutes or so before giving up. The build command has -allConfigurations so it builds all product+test assemblies for all platforms ({linux,win,osx,freebsd,illumos}-{x86,x64,arm,arm64,riscv64.. etc.}) all in one invocation of build (which isn't exactly efficient as we should probably group them..), which means, as it stands, this leg needs decent amount of RAM.

steveisok · 2024-10-07T23:32:59Z

@ilyas1974 @markwilkie I don't think our RAM consumption has increased that much to not be able to handle this configuration. Thoughts?

ericstj · 2024-10-10T17:47:41Z

I took a look at a few passing builds, many have the same logs where they approach 95% memory usage but eventually succeed.

Here are some samples:
https://dev.azure.com/dnceng-public/public/_build/results?buildId=836601&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857
https://dev.azure.com/dnceng-public/public/_build/results?buildId=838110&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857&l=3923
https://dev.azure.com/dnceng-public/public/_build/results?buildId=837745&view=logs&j=e80acbf0-bc87-577c-4c46-0016b0794913&t=f0fa9d72-e49a-5249-4d28-1199014b9857&l=4228

Near this point I see logs like this:

  initializing ChangeMakerService with capabilities: Baseline, AddMethodToExistingType, AddStaticFieldToExistingType, AddInstanceFieldToExistingType, NewTypeDefinition, ChangeCustomAttributes, UpdateParameters, GenericAddMethodToExistingType, GenericUpdateMethod, GenericAddFieldToExistingType
  baseline ready
  got a change
  parsing patch #1 from /__w/1/s/src/libraries/System.Runtime.Loader/tests/ApplyUpdate/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField/GenericAddInstanceField_v1.cs and creating delta
  Found changes in GenericAddInstanceField.cs
  change service made fa564b82-cf1c-4fb0-9d1a-f5ca4c71ff03
  wrote /__w/1/s/artifacts/bin/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField/Debug/net10.0/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField.dll.1.dmeta
  got a change
  parsing patch #2 from /__w/1/s/src/libraries/System.Runtime.Loader/tests/ApplyUpdate/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField/GenericAddInstanceField_v2.cs and creating delta
  Found changes in GenericAddInstanceField.cs
  change service made fa564b82-cf1c-4fb0-9d1a-f5ca4c71ff03
  wrote /__w/1/s/artifacts/bin/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField/Debug/net10.0/System.Reflection.Metadata.ApplyUpdate.Test.GenericAddInstanceField.dll.2.dmeta
  done

It looks to me like this is coming from https://github.com/dotnet/hotreload-utils/blob/254ec75de6127c368827d15c3af2477095b8b1b4/src/Microsoft.DotNet.HotReload.Utils.Generator/EnC/ChangeMakerService.cs#L28

Does anyone have an idea why hotreload would be running during a build?? I could imagine that if some hot reload service was runnign during a build or if tests were running while the product was building that could explain high memory usage.

lewing added First Responder Known Build Error labels Aug 26, 2024

guihkx mentioned this issue Aug 26, 2024

"Error: 'NoneType' object is not subscriptable" on Channel.from_directory DevilXD/TwitchDropsMiner#547

Closed

jkotas mentioned this issue Aug 26, 2024

Fix alloc-dealloc mismatch dotnet/runtime#106957

Merged

github-staff deleted a comment from masooddahmedd Sep 10, 2024

Maoni0 mentioned this issue Sep 16, 2024

instrumentation (mostly) for DATAS dotnet/runtime#107853

Merged

lewing mentioned this issue Sep 17, 2024

Fix RegexOptions.Compiled|IgnoreCase perf when dynamic code isn't supported dotnet/runtime#107874

Merged

CarnaViire mentioned this issue Sep 18, 2024

WebSocket Feedback Follow-up dotnet/runtime#107662

Merged

am11 mentioned this issue Oct 7, 2024

Disable linux-x64 dev-innerloop dotnet/runtime#108581

Merged

am11 mentioned this issue Oct 7, 2024

Move more helpers from DomainAssembly to Assembly dotnet/runtime#108568

Merged

This was referenced Oct 7, 2024

[main] Update dependencies from 7 repositories dotnet/runtime#108385

Merged

[main] Update dependencies from dotnet/arcade dotnet/runtime#108323

Merged

[main] Update dependencies from dotnet/source-build-reference-packages dotnet/runtime#108593

Merged

am11 mentioned this issue Oct 8, 2024

[RISC-V] Fix some minor errors dotnet/runtime#107179

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

restarted. Azure DevOps can't recover from restarts. #3879

restarted. Azure DevOps can't recover from restarts. #3879

lewing commented Aug 26, 2024 •

edited by build-analysis bot

Loading

am11 commented Oct 7, 2024

am11 commented Oct 7, 2024

steveisok commented Oct 7, 2024

ericstj commented Oct 10, 2024

restarted. Azure DevOps can't recover from restarts. #3879

restarted. Azure DevOps can't recover from restarts. #3879

Comments

lewing commented Aug 26, 2024 • edited by build-analysis bot Loading

Build

Build leg reported

Pull Request

Known issue core information

Release Note Category

Release Note Description

Additional information about the issue reported

Known issue validation

Report

Summary

am11 commented Oct 7, 2024

am11 commented Oct 7, 2024

steveisok commented Oct 7, 2024

ericstj commented Oct 10, 2024

lewing commented Aug 26, 2024 •

edited by build-analysis bot

Loading