Official runtime builds for .NET 7 and 8 frequently failing during artifact upload

[mmitche]

Build

https://dev.azure.com/dnceng/internal/_build/results?buildId=2141349&view=logs&j=4d50a8bf-a143-51c7-5cc8-defff437e23b&t=0b0b242f-bbcb-57b5-fe9f-26dc042642ec&l=8624

Build leg reported

Prepare artifacts and upload to build

Pull Request

No response

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

• Open the failing build above and investigate
• Add a comment explaining your findings

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

1. Add the label "Known Build Error"
2. Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation

{
   "ErrorMessage" : "A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (dev.azure.com:443)",
   "BuildRetry": false,
   "ErrorPattern": "",
   "ExcludeConsoleLog": false
}

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

This seems to appear only during artifact upload after signing has completed. The machine's connection to AzDO is getting throttled.

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=316909
Error message validated: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (dev.azure.com:443)
Result validation: ❌ Known issue did not match with the provided build.
Validation performed at: 6/28/2023 9:03:47 PM UTC

[mmitche]

https://dev.azure.com/dnceng/internal/_build/results?buildId=2142911&view=results
https://dev.azure.com/dnceng/internal/_build/results?buildId=2142794&view=results
https://dev.azure.com/dnceng/internal/_build/results?buildId=2142788&view=results
https://dev.azure.com/dnceng/internal/_build/results?buildId=2141994&view=results
https://dev.azure.com/dnceng/internal/_build/results?buildId=2141900&view=results

Appears to be hitting 30% of the time?

[mmitche]

This appears to be because of server side throttling during artifact upload:

##[warning]The job is currently being throttled by the server. You may experience delays in console line output, job status reporting, and task log uploads.
,##[warning]Link to resource utilization page (global 1-hour view): https://dev.azure.com:443/dnceng/_usersSettings/usage?tab=pipelines&queryDate=2023-03-13T17%3a57%3a33%2c2023-03-13T18%3a57%3a33.
Link to resource utilization page (1-hour view by pipeline): https://dev.azure.com:443/dnceng/_usersSettings/usage?tab=pipelines&queryDate=2023-03-13T17%3a57%3a33%2c2023-03-13T18%3a57%3a33&keywords=20230313.4&definition=dotnet-runtime-official.
,##[warning]The job has experienced 85.467 seconds total delay caused by server throttling.
Pool: [NetCore1ESPool-Internal](https://dev.azure.com/dnceng/7ea9116e-9fac-403d-b258-b31fcf1bb293/_settings/agentqueues?poolId=&queueId=394)
Agent: NetCore1ESPool-Internal 1
Started: Mar 13 at 11:41 AM
Duration: 22m 32s

[mmitche]

First time this shows up is March 10th: https://dev.azure.com/dnceng/internal/_build/results?buildId=2132930&view=results

[missymessa]

@mmitche do we want to escalate this to AzDO in an IcM or is this something we need to change in our code to alleviate the throttling?

[mmitche]

I think this is an AzDO side issue. We're just issueing the upload commands

[missymessa]

Probably enough of an impact that it warrants a Sev3 IcM?

[mmitche]

Yep. Nothing that requires us to sit on a bridge on Friday though :)

[epananth]

Created ICM https://portal.microsofticm.com/imp/v3/incidents/details/377257323/home

[ilyas1974]

@mmitche, based on the metrics I'm seeing above, this issue is not as widespread as it may have been last month. I wanted to touch base with you and see how you would like to proceed here. We have not gotten much traction on our ICM and don't really have current data available for additional investigation.

Recommendation - close this issue and the ICM for now. If this issue occurs again, we can revisit our interact with the AzDO team.

[mmitche]

Seeing this again: https://dev.azure.com/dnceng/internal/_build/results?buildId=2169006&view=logs&j=4d50a8bf-a143-51c7-5cc8-defff437e23b&t=0b0b242f-bbcb-57b5-fe9f-26dc042642ec&l=7199

[mmitche]

A bunch of recent runtime builds are seeing it: https://dev.azure.com/dnceng/internal/_build/results?buildId=2169253&view=results (attempt 1)