Error reported in diagnostic logs. Please examine the log for more details

[jeffschwMSFT]

Build

https://dev.azure.com/dnceng/internal/_build/results?buildId=2367532

Error:

System.Net.Http.HttpRequestException: No route to host (dev.azure.com:443)
 ---> System.Net.Sockets.SocketException (65): No route to host
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|277_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Microsoft.VisualStudio.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Location.Client.LocationHttpClient.GetConnectionDataAsync(ConnectOptions connectOptions, Int64 lastChangeId, CancellationToken cancellationToken, Object userState)
   at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.GetConnectionDataAsync(ConnectOptions connectOptions, Int32 lastChangeId, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.Location.VssServerDataProvider.ConnectAsync(ConnectOptions connectOptions, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Agent.TaskServer.ConnectAsync(VssConnection jobConnection) in /Users/runner/work/1/s/src/Microsoft.VisualStudio.Services.Agent/TaskServer.cs:line 42
   at Microsoft.VisualStudio.Services.Agent.Worker.JobRunner.RunAsync(AgentJobRequestMessage message, CancellationToken jobRequestCancellationToken) in /Users/runner/work/1/s/src/Agent.Worker/JobRunner.cs:line 224
   at Microsoft.VisualStudio.Services.Agent.Worker.JobRunner.RunAsync(AgentJobRequestMessage message, CancellationToken jobRequestCancellationToken) in /Users/runner/work/1/s/src/Agent.Worker/JobRunner.cs:line 397
   at Microsoft.VisualStudio.Services.Agent.Worker.Worker.RunAsync(String pipeIn, String pipeOut) in /Users/runner/work/1/s/src/Agent.Worker/Worker.cs:line 97
   at Microsoft.VisualStudio.Services.Agent.Worker.Program.MainAsync(IHostContext context, String[] args) in /Users/runner/work/1/s/src/Agent.Worker/Program.cs:line 52

Error reported in diagnostic logs. Please examine the log for more details

Build leg reported

No response

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "Error reported in diagnostic logs. Please examine the log for more details",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2367532
Error message validated: Error reported in diagnostic logs. Please examine the log for more details
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/1/2024 3:50:06 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2932894	dotnet-maui	Build Device Tests (Mono)
2932475	dotnet-maui	Build Device Tests (Mono)
2932022	dotnet-dotnet-buildtools-prereqs-docker	Linux_amd64 src-azurelinux-3.0-net11.0-android-graph
2931936	dotnet-maui	Controls (v18.5) Shell	Log
2931849	dotnet-maui	Controls (v18.5)	Log
2931579	dotnet-dotnet-buildtools-prereqs-docker	Linux_amd64 src-azurelinux-3.0-net11.0-android-graph	Log
2931846	dotnet-maui	Build Device Tests (Mono)
2931698	dotnet-maui	Controls (vlatest) CarouselView	Log
2931543	dotnet-maui	Controls (vlatest) SafeAreaEdges,Shadow	Log
2931333	dotnet-dotnet-buildtools-prereqs-docker	Linux_amd64 src-azurelinux-3.0-net9.0-android-graph
2930751	dotnet-maui	Controls (vlatest) Editor,Effects,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView	Log
2930970	dotnet-maui	Controls (v18.5)	Log
2930951	dotnet-runtime	browser-wasm linux release Mono
2930819	dotnet-maui	Controls (vlatest) Page,Performance,Picker,ProgressBar	Log
2930821	dotnet-runtime	browser-wasm linux release Mono_multithread multithread
2930755	dotnet-dotnet-buildtools-prereqs-docker	Linux_amd64 src-azurelinux-3.0-net10.0-android-graph	Log
2930592	dotnet-dotnet-buildtools-prereqs-docker	Linux_amd64 src-azurelinux-3.0-net10.0-android-graph
2930261	dotnet-runtime	browser-wasm linux release Mono_multithread multithread
2930194	dotnet-runtime	linux-arm64 Debug Libraries_CheckedCoreCLR		#57892
2930080	dotnet-runtime	linux-arm64 Debug Libraries_CheckedCoreCLR		#58741
2929887	dotnet-maui	Controls (v18.4) ListView	Log
2929956	dotnet-dotnet-buildtools-prereqs-docker	Linux_amd64 src-azurelinux-3.0-net10.0-android-graph
2929942	dotnet-runtime	linux-arm64 Debug Mono_MiniJIT_LibrariesTests		#59204
2929919	dotnet-runtime	browser-wasm linux Release LibraryTests_EAT		#59204
2929781	dotnet-dotnet-buildtools-prereqs-docker	Linux_amd64 src-azurelinux-3.0-net11.0-android-graph
2929742	dotnet-maui	Controls (v18.4) CollectionView	Log
2929634	dotnet-dotnet-buildtools-prereqs-docker	Linux_amd64 src-azurelinux-3.0-net10.0-android-graph
2929520	dotnet-runtime	linux-arm64 release coreclr
2928850	dotnet-runtime	linux-arm64 Debug Mono_MiniJIT_LibrariesTests	Log	#57892
2928707	dotnet-runtime	browser-wasm linux release bluebird
2928524	dotnet-maui	Controls (v18.4) Editor,Effects,FlyoutPage,Focus,Fonts,Frame,Gestures,GraphicsView
2928241	dotnet-dotnet-buildtools-prereqs-docker	Linux_amd64 src-azurelinux-3.0-net10.0-android-graph
2928202	dotnet-runtime	browser-wasm linux release bluebird
2928182	dotnet-runtime	linux-arm64 release AOT
2928183	dotnet-runtime	browser-wasm linux release Mono_multithread multithread
2928168	dotnet-runtime	linux-x64 Debug Mono_MiniJIT_LibrariesTests		#57892
2928143	dotnet-runtime	linux-x64 release coreclr
2928111	dotnet-runtime	linux-arm64 release AOT
2928043	dotnet-dotnet-buildtools-prereqs-docker	Linux_amd64 src-azurelinux-3.0-net10.0-android-graph
2928051	dotnet-runtime	linux-x64 release coreclr
2928094	dotnet-runtime	linux-x64 release AOT
2928045	dotnet-runtime	linux-arm64 release AOT
2924275	dotnet-runtime	linux-x64 release coreclr_r2r_interpreter
2928024	dotnet-runtime	browser-wasm linux Release wasm
2928009	dotnet-runtime	linux-x64 release AOT
2927977	dotnet-runtime	browser-wasm linux Release wasm
2927962	dotnet-runtime	linux-arm64 release AOT
2927847	dotnet-dotnet-buildtools-prereqs-docker	Linux_amd64 src-azurelinux-3.0-net11.0-android-graph
2927944	dotnet-runtime	browser-wasm linux Release wasm
2927900	dotnet-runtime	linux-arm64 release AOT
2927875	dotnet-runtime	linux-arm64 release AOT
2927857	dotnet-runtime	linux-x64 release coreclr
2927854	dotnet-runtime	browser-wasm linux Release wasm_coreclr
2927648	dotnet-maui	Controls (v18.4) Navigation
2927836	dotnet-runtime	linux-x64 release bluebird
2927818	dotnet-runtime	browser-wasm linux Release wasm
2927790	dotnet-runtime	browser-wasm linux Release wasm_coreclr
2927718	dotnet-dotnet-buildtools-prereqs-docker	Linux_amd64 src-azurelinux-3.0-net11.0-android-graph
2927758	dotnet-runtime	linux-arm64 release AOT
2927747	dotnet-runtime	linux-x64 release AOT
2927580	dotnet-maui	Controls (v18.4) TabbedPage,TableView,TimePicker,TitleView,ToolbarItem	Log
2927728	dotnet-runtime	linux-x64 release AOT
2927676	dotnet-runtime	linux-x64 release CodeQL
2927640	dotnet-runtime	linux-x64 release AOT
2927631	dotnet-runtime	linux-x64 release AOT
2927607	dotnet-runtime	linux-x64 release AOT
2927606	dotnet-runtime	linux-x64 release AOT
2927544	dotnet-dotnet-buildtools-prereqs-docker	Linux_amd64 src-azurelinux-3.0-net8.0-android-graph
2927515	dotnet-node	linux-musl arm64	Log
2927465	dotnet-runtime	browser-wasm linux Release wasm
2927364	dotnet-runtime	linux-arm64 release AOT
2927353	dotnet-runtime	browser-wasm linux Release wasm
2927354	dotnet-runtime	linux-arm64 release AOT
2927291	dotnet-runtime	browser-wasm linux Release wasm
2927276	dotnet-runtime	linux-arm64 release
2927215	dotnet-dotnet-buildtools-prereqs-docker	Linux_amd64 src-azurelinux-3.0-net10.0-android-graph
2927121	dotnet-dotnet-buildtools-prereqs-docker	Linux_amd64 src-azurelinux-3.0-net11.0-android-graph
2927142	dotnet-runtime	linux-arm64 release
2927086	dotnet-runtime	linux-x64 release coreclr
2927085	dotnet-runtime	linux-x64 release coreclr
2927018	dotnet-runtime	linux-arm64 release AOT
2926962	dotnet-runtime	linux-x64 release coreclr
2926891	dotnet-dotnet-buildtools-prereqs-docker	Linux_amd64 src-azurelinux-3.0-net9.0-crossdeps-graph	Log
2926928	dotnet-runtime	browser-wasm linux Release wasm_coreclr
2926895	dotnet-runtime	linux-x64 release coreclr
2926894	dotnet-runtime	linux-arm64 release AOT
2926773	dotnet-dotnet-buildtools-prereqs-docker	Linux_amd64 src-azurelinux-3.0-net11.0-android-graph
2926782	dotnet-runtime	linux-x64 release coreclr
2926750	dotnet-runtime	linux-arm release
2926738	dotnet-runtime	linux-x64 release coreclr
2926621	dotnet-runtime	browser-wasm linux Release wasm_coreclr
2926620	dotnet-runtime	browser-wasm linux Release wasm
2926603	dotnet-runtime	linux-arm64 release AOT
2926593	dotnet-runtime	Libraries SuperPMI collection libraries_tests Checked coreclr osx arm64 Release
2926505	dotnet-runtime	browser-wasm linux Release wasm
2926543	dotnet-runtime	wasi-wasm linux release CodeQL
2926508	dotnet-runtime	browser-wasm linux Release AllSubsets_Mono_RuntimeTests monointerpreter
2926495	dotnet-runtime	linux-arm64 release AOT
2926467	dotnet-runtime	linux-arm64 release coreclr
2926451	dotnet-runtime	linux-x64 release AOT
Displaying 100 of 193 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
1	76	192

[lewing]

Looks like mostly macs? not sure where Build Source Index runs

[epananth]

Looked at the latest builds all are failing cos out of space on the device

[epananth]

##[error]Error reported in diagnostic logs. Please examine the log for more details. - /Users/runner/runners/3.242.1/_diag/Worker_20240806-221514-utc.log System.IO.IOException: No space left on device : '/Users/runner/runners/3.242.1/_diag/Worker_20240806-221514-utc.log' at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan1 buffer, Int64 fileOffset)
at System.IO.Strategies.BufferedFileStreamStrategy.FlushWrite()
at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
at System.Diagnostics.TextWriterTraceListener.Flush()
at Microsoft.VisualStudio.Services.Agent.HostTraceListener.WriteHeader(String source, TraceEventType eventType, Int32 id) in /Users/runner/work/1/s/src/Microsoft.VisualStudio.Services.Agent/HostTraceListener.cs:line 151
at Microsoft.VisualStudio.Services.Agent.HostTraceListener.TraceEvent(TraceEventCache eventCache, String source, TraceEventType eventType, Int32 id, String message) in /Users/runner/work/1/s/src/Microsoft.VisualStudio.Services.Agent/HostTraceListener.cs:line 81
at System.Diagnostics.TraceSource.TraceEvent(TraceEventType eventType, Int32 id, String message)
at Microsoft.VisualStudio.Services.Agent.Worker.Worker.RunAsync(String pipeIn, String pipeOut) in /Users/runner/work/1/s/src/Agent.Worker/Worker.cs:line 93
at Microsoft.VisualStudio.Services.Agent.Worker.Program.MainAsync(IHostContext context, String[] args) in /Users/runner/work/1/s/src/Agent.Worker/Program.cs:line 52`

[epananth]

I am going to go ahead an add a check for disk utilization, to know more why this is failing.

[epananth]

Is it possible to add this check to your pipeline to get more data (like aspnet-core team did)

https://github.com/dotnet/aspnetcore/blob/42af9fe6ddd7c3f9cde04ac003bf97509881873b/.azure/pipelines/jobs/default-build.yml#L163-L165

[missymessa]

@jeffschwMSFT @lewing is this still an issue?

[lewing]

@jeffschwMSFT @lewing is this still an issue?

yes

[lewing]

I took a look at the recent entries in this issue. All the latest builds hitting this error are from the dotnet-runtime-perf and dotnet-runtime-perf-build pipelines running on NetCore1ESPool-Internal agents.

Checking the build logs (e.g. builds 2926621, 2926620, 2926505, 2926495, 2926451, 2926370), they all show the same root cause: disk space exhaustion on the agents — consistently at 97–100% usage. This triggers a cascade of failures:

• Docker pulls fail with "no space left on device"
• Build processes crash with SIGABRT (exit code 134)
• Timeouts as processes stall on disk I/O
• The generic "Error reported in diagnostic logs" wrapper error

This appears to be the same underlying issue reported by @epananth back in Aug 2024 — the perf build agents need either larger disks or more aggressive cleanup between jobs. The disk utilization check that was added helps surface the problem, but the root cause (insufficient disk capacity for the workload) is still present.

[lewing]

cc @DrewScoggins @mangod9

[DrewScoggins]

Yeah, we are seeing this as well. We have a PR up, dotnet/runtime#125634, to fix this on the build side, and should have another up to fix the perf job side as well.