[8.0.400] MSBuild Crash when publishing containers

[baronfel]

In 8.0.400, we added some telemetry (documented here) for container publish to help us direct investment in the tech. This telemetry is reported through MSBuild's built-in telemetry hooks, which for the dotnet CLI ends up forwarding telemetry from MSBuild through the CLI Telemetry's configuration (aka the environment variable mentioned above: DOTNET_CLI_TELEMETRY_OPTOUT).

There was a bug in the MSBuild engine's handling of this telemetry data, which we fixed in this PR, and this bug is the cause of the crash that's reported. We found this error during final testing, but too late to include it in the 400 release, and so began the process of making a hotfix immediately. This hotfix, 8.0.401, should be available in the next day or so.

The main reason we decided to do a hotfix for this is that the proposed workaround - setting DOTNET_CLI_TELEMETRY_OPTOUT to 1/true/other truthy values - did not consistently work in broader testing. This is because MSBuild is a multi-node system, and telemetry events generated on a worker node must be serialized and forwarded to the central node. This serialization is where the bug lies. If you can guarantee that MSBuild operates in a single-node mode (via /m:1 or similar) that may work as a workaround as well, but several high-impact use cases like azd deployment do not expose the configuration necessary to ensure this behavior.

Other reports:

• Dotnet Publish crashes with error 'MSB4166: Child node "##" exited prematurely. Shutting down.' actions/runner-images#10437
• Dotnet publish container regression in 8.0.400 sdk-container-builds#587
• [Broken Build]: MSB4166 after upgrading to SDK 8.0.400: 'Child node "2" exited prematurely' msbuild#10518
• Dotnet publish container regression in 8.0.400 #42719

[baronfel]

The 8.0.401 hotfix SDK has been released on all channels and contains a fix for this issue.

[mip1983]

@baronfel I originally tried 8.0.400 and got this container issue. I tried 8.0.401 this morning but the CI/CD pipeline seemed to break at the package restore stage with errors like:

"C:\hostedtoolcache\windows\dotnet\sdk\8.0.401\NuGet.targets(174,5): error : The feed 'nuget.org [https://api.nuget.org/v3/index.json]' lists package 'Microsoft.AspNetCore.Components.Authorization.8.0.8' but multiple attempts to download the nupkg have failed. The feed is either invalid or required packages were removed while the current operation was in progress. Verify the package exists on the feed and try again. [D:\a\1\s\ecoDriverAnalytics\ecoDriverAnalytics.Services.Tests\ecoDriverAnalytics.Services.Tests.csproj]
Failed to restore D:\a\1\s\ecoDriverAnalytics\ecoDriverAnalytics.ApiClient\ecoDriverAnalytics.Api.Client.csproj (in 625 ms)."

Reverting back to 8.0.303 seems to fix it, is this in some way related?

[baronfel]

My best guess is that this is a connectivity problem unique to your environment somehow. The package is available on nuget.org and has many downloads at this time.

[metoule]

For what it's worth, we're also seeing a lot of these errors after the update to 8.0.401

The feed nuget.org lists package 'whatever' but multiple attempts to download the nupkg have failed. 

The package that fails seems random.

[AntoineBichonBroadsign]

Same for us, since the 8.0.4xx our ci/cd pipeline (via circleci) randomly fail on dotnet restore. We will revert to 8.0.304

[baronfel]

Hey folks - restore errors are entirely different from the container publishing system, please raise restore errors over at the NuGet/Home repo where the NuGet team will see them.

[metoule]

Thanks for the heads up, I've created NuGet/Home#13729