Linux musl x64 build break apparently due to some libc version mismatch (high priority as it is blocking clean CI)

[trylek]

Platform: Linux musl x64
Example run: https://dev.azure.com/dnceng-public/public/_build/results?buildId=241777&view=logs&j=92e17f34-a5d5-5d2c-e1e5-826eb9786835&t=8b99f496-81fe-5fa6-0e3e-125347621b1c

Diagnostics:

  249 / 264 (94%, 249 failed): failed in 667 msecs, exit code 134, expected 0: dotnet /__w/1/s/artifacts/bin/coreclr/linux.x64.Checked/crossgen2/crossgen2.dll @/__w/1/s/artifacts/tests/coreclr/obj/linux.x64.Checked/crossgen.out/System.Runtime.Loader.dll.rsp
  !! Unhandled exception. System.DllNotFoundException: Unable to load shared library 'jitinterface_x64' or one of its dependencies. In order to help diagnose loading problems, consider using a tool like strace. If you're using glibc, consider setting the LD_DEBUG environment variable: 
  !! /__w/1/s/.dotnet/shared/Microsoft.NETCore.App/8.0.0-preview.3.23174.8/jitinterface_x64.so: cannot open shared object file: No such file or directory
  !! /__w/1/s/artifacts/bin/coreclr/linux.x64.Checked/crossgen2/jitinterface_x64.so: cannot open shared object file: No such file or directory
  !! /__w/1/s/.dotnet/shared/Microsoft.NETCore.App/8.0.0-preview.3.23174.8/libjitinterface_x64.so: cannot open shared object file: No such file or directory
  !! libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory
  !! /__w/1/s/.dotnet/shared/Microsoft.NETCore.App/8.0.0-preview.3.23174.8/jitinterface_x64: cannot open shared object file: No such file or directory
  !! /__w/1/s/artifacts/bin/coreclr/linux.x64.Checked/crossgen2/jitinterface_x64: cannot open shared object file: No such file or directory
  !! /__w/1/s/.dotnet/shared/Microsoft.NETCore.App/8.0.0-preview.3.23174.8/libjitinterface_x64: cannot open shared object file: No such file or directory
  !! /__w/1/s/artifacts/bin/coreclr/linux.x64.Checked/crossgen2/libjitinterface_x64: cannot open shared object file: No such file or directory
  !!    at System.Runtime.InteropServices.NativeLibrary.LoadByName(String libraryName, QCallAssembly callingAssembly, Boolean hasDllImportSearchPathFlag, UInt32 dllImportSearchPathFlag, Boolean throwOnError)
  !!    at System.Runtime.InteropServices.NativeLibrary.LoadLibraryByName(String libraryName, Assembly assembly, Nullable`1 searchPath, Boolean throwOnError)
  !!    at Internal.JitInterface.JitConfigProvider.<>c__DisplayClass5_0.b__0(String libName, Assembly assembly, Nullable`1 searchPath) in /_/src/coreclr/tools/Common/JitInterface/JitConfigProvider.cs:line 59
  !!    at System.Runtime.InteropServices.NativeLibrary.LoadLibraryCallbackStub(String libraryName, Assembly assembly, Boolean hasDllImportSearchPathFlags, UInt32 dllImportSearchPathFlags)
  !!    at Internal.JitInterface.CorInfoImpl.GetJitHost(IntPtr configProvider)
  !!    at Internal.JitInterface.CorInfoImpl.Startup(CORINFO_OS os) in /_/src/coreclr/tools/Common/JitInterface/CorInfoImpl.cs:line 179
  !!    at Internal.JitInterface.JitConfigProvider.Initialize(TargetDetails target, IEnumerable`1 jitFlags, IEnumerable`1 parameters, String jitPath) in /_/src/coreclr/tools/Common/JitInterface/JitConfigProvider.cs:line 64
  !!    at ILCompiler.ReadyToRunCodegenCompilationBuilder.ToCompilation() in /_/src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/ReadyToRunCodegenCompilationBuilder.cs:line 296
  !!    at ILCompiler.Program.RunSingleCompilation(Dictionary`2 inFilePaths, InstructionSetSupport instructionSetSupport, String compositeRootPath, Dictionary`2 unrootedInputFilePaths, HashSet`1 versionBubbleModulesHash, ReadyToRunCompilerContext typeSystemContext) in /_/src/coreclr/tools/aot/crossgen2/Program.cs:line 613
  !!    at ILCompiler.Program.Run() in /_/src/coreclr/tools/aot/crossgen2/Program.cs:line 289
  !!    at ILCompiler.Crossgen2RootCommand.<>c__DisplayClass187_0.<.ctor>b__0(InvocationContext context) in /_/src/coreclr/tools/aot/crossgen2/Crossgen2RootCommand.cs:line 293
  !!    at System.CommandLine.Invocation.AnonymousCommandHandler.Invoke(InvocationContext context)
  !!    at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<b__0>d.MoveNext()
  !! --- End of stack trace from previous location ---
  !!    at System.CommandLine.CommandLineBuilderExtensions.<>c__DisplayClass16_0.<b__0>d.MoveNext()
  !! --- End of stack trace from previous location ---
  !!    at System.CommandLine.CommandLineBuilderExtensions.<>c__DisplayClass11_0.<b__0>d.MoveNext()
  !! --- End of stack trace from previous location ---
  !!    at System.CommandLine.CommandLineBuilderExtensions.<>c__DisplayClass22_0.<b__0>d.MoveNext()
  !! --- End of stack trace from previous location ---
  !!    at System.CommandLine.Invocation.InvocationPipeline.g__FullInvocationChain|3_0(InvocationContext context)
  !!    at System.CommandLine.Invocation.InvocationPipeline.Invoke(IConsole console)
  !!    at System.CommandLine.Parsing.ParseResultExtensions.Invoke(ParseResult parseResult, IConsole console)
  !!    at System.CommandLine.Parsing.ParserExtensions.Invoke(Parser parser, String[] args, IConsole console)
  !!    at ILCompiler.Program.Main(String[] args) in /_/src/coreclr/tools/aot/crossgen2/Program.cs:line 890

/cc @janvorli @sbomer @dotnet/runtime-infrastructure @dotnet/crossgen-contrib

Known Issue Error Message

Fill the error message using known issues guidance.

{
  "ErrorMessage": "Unable to load shared library 'jitinterface_x64'",
  "BuildRetry": false
}

Report

Build	Definition	Step Name	Console log	Pull Request
241777	dotnet/runtime	Generate CORE_ROOT	Log
241282	dotnet/runtime	Generate CORE_ROOT	Log
241126	dotnet/runtime	Generate CORE_ROOT	Log
241070	dotnet/runtime	Generate CORE_ROOT	Log	#84904

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
3	4	4

[janvorli]

The error sounds as if we attempted to load MUSL version of the libjitinterface_x64.so on a glibc distro.

[janvorli]

@sbomer It looks like the cross build is attempting to use crossgen2 with MUSL version of JIT native libraries while it should use the glibc ones.

[trylek]

@janvorli - I see, so am I right to understand that in this particular case HostArchitecture=Linux whereas TargetArchitecture=Linux_musl and some code publishing the JIT native libraries uses the TargetArchitecture version instead of the HostArchitecture which should be used instead as that's the OS on which the build is running?

[trylek]

/cc @dotnet/jit-contrib

[sbomer]

I see this in the logs:

--crossgen2-path "/__w/1/s/artifacts/bin/coreclr/linux.x64.Checked/crossgen2/crossgen2.dll"

The musl x64 build changed to become a cross-build (build on x64 glibc, targeting x64 musl), so I think this needs to be changed to use the x64 glibc hosted crossgen2 that gets built as part of the runtime cross components build. I think we need to address this TODO:

runtime/src/tasks/Crossgen2Tasks/Microsoft.NET.CrossGen.targets

Lines 76 to 78 in add51b8

	<CrossgenDir>$([System.IO.Path]::Combine($(_CoreclrPkgDir),"tools"))</CrossgenDir>
	<!-- TODO override with rid specific tools path for x-arch -->
	<Crossgen>$([System.IO.Path]::Combine($(CrossgenDir),"crossgen"))</Crossgen>

[sbomer]

For example here's how we handle this when we crossgen corelib:

runtime/src/coreclr/crossgen-corelib.proj

Line 16 in add51b8

<CrossDir Condition="'$(CrossBuild)' == 'true' or '$(BuildArchitecture)' != '$(TargetArchitecture)'">$(BuildArchitecture)</CrossDir>

runtime/src/coreclr/crossgen-corelib.proj

Line 78 in add51b8

<CrossGenDllCmd>$(DotNetCli) $([MSBuild]::NormalizePath('$(BinDir)', '$(CrossDir)', 'crossgen2', 'crossgen2.dll'))</CrossGenDllCmd>

[sbomer]

I will try to reproduce this locally, but if someone is already set up to run the R2R outerloop tests locally, it might help us to fix and validate this faster.

[sbomer]

Actually, I think the place that needs to be fixed up is

runtime/src/tests/build.proj

Lines 529 to 532 in add51b8

	<CrossgenDir>$(__BinDir)</CrossgenDir>
	<CrossgenDir Condition="'$(TargetArchitecture)' == 'arm'">$(CrossgenDir)\x64</CrossgenDir>
	<CrossgenDir Condition="'$(TargetArchitecture)' == 'arm64'">$(CrossgenDir)\x64</CrossgenDir>
	<CrossgenDir Condition="'$(TargetArchitecture)' == 'x86'">$(CrossgenDir)\x64</CrossgenDir>

(not the Crossgen2Tasks targets)

[trylek]

Overall I think this is the right place to fix this as this constructs the path to Crossgen2 for compiling the framework but I'm still somewhat struggling with understanding how this pre-existing cross-architecture scheme translates into cross-OS-variation build. If I'm right to understand JanV's comments, at some point we're likely producing separate Linux-targeting vs. Linux-musl-targeting jitinterface and we're loading / using the wrong one. I'll need to remind myself how we're producing Crossgen2 versions targeting these OS variants before I can start thinking about a fix.

[trylek]

I have asked @davidwrighton offline for help as I believe he wrote most of these scripts, I'll continue working on this tomorrow if we don't nail it down before EOD today. One thing I'm still missing is why has this blown up right now considering we've been running Linux-musl tests for years as far as I recall?