ARM64 and cross compile support

[RalfKornmannEnvision]

About a year a year ago I was working on an evaluation to use CoreRT to port the C# part of PC/Windows based game to they major game consoles (XBox One; PlayStation 4 and Switch). The Team had it already running with Mono but was not satisfied with the overall performances. Therefore I was tasked with checking if the performances can be improved or finding an alternative to using Mono. 

As getting CoreRT working on the XBox was an very easy task I was able to get the core module of our C# code run in a few hours. As it runs more than twice as fast compared to Mono it was decided to go ahead. PlayStation was a little bit more work but it worked after a few days and showed the same performances improvements. Unfortunately while goingon to the Switch I noticed that the ARM64 support was incomplete and using ARM was not an option because of some limitations on the target system. As the team still wanted the performances improvements I started to implement the missing parts. It took some time but I was able to get our test case running and again it was more than twice as fast as the Mono version. It does not use all .Net features therefore there are still parts missing. But as our test test case (around 5MB of C# code) was able to run for hours with the precise GC I assume it is at least a big part.

Unfortunately priorities changed and this project was put on hold. After things have changed again it was decided to give a switch from Mono to CoreRT a second try. This time I although got the permission to submit anything that does not touch the various NDAs. As you might assume I needed to add code on different places including the jit itself. As anything was done on a now nearly one year old version I need to reintegrate all these changes again. But I'd like to make sure they can easily go back in the main code branch. 

Therefore the question on which of these changes you are interested in and what's your recommended work flow?

[MichalStrehovsky]

Awesome, thank you!

I'll defer to @jkotas for final word, but my take:

• We should take RyuJIT changes directly to dotnet/runtime master branch. That's where the rest of CoreRT RyuJIT changes live and I assume it's not an extensive change; just tweaks. We'll either move them later to be exclusive to runtimelab or just leave them there.
• I would be happy to review and merge the ILCompiler changes here as you have them.
• Same with whatever runtime changes you have.
• When it comes to the cross compilation build system changes, we should just wait until CoreRT repo is moved over into runtimelab repo (Move native AOT experiment to dotnet/runtimelab runtimelab#4) - we're going to hook CoreRT build up into the CoreCLR build there and that already handles crosscompilation (it's probably still similar to how CoreRT crosscompiles, but CoreCLR has been evolving and CoreRT has been frozen in time for a while). If the changes you have are already good to merge, we could merge them, but crossbuild will have to be hooked up again in the runtimelab repo once we finish the move (so... probably no need to polish it if it's not polished).

[RalfKornmannEnvision]

I am still in the process of setting up my new development PC so I might remember some things wrong.

IIRC the RyuJIT changes were primary the implementation of the ARM64 unwinding information that the GC needs for the stack walk. They were just missing in the version that was used at that time. 

I needed to build a new ObjectWriter as the default one was not able to write ARM64 for ELF code at all. Beside of this I compiled a version that could write all the different binary formats for all the CPUs I need.
The changes in the ILCompiler itself were mostly the parts were these small pieces of assembler code is generated to glue things together.

I have some platform independent changes for the coreRT runtime itself, too. Mostly the implementation of missing ARM64 assembler functions and getting the stackwalk working. Unfortunately most of the other runtime changes are platform specific and therefore I can not make them public. We might be able to talk about the XBox One parts if Microsoft is interested to give other XBox developers access to it.

The cross compiler support was done very simple. Just using different RyuJit dlls depending on the target settings and using an own project file for each platform that contains the links to the platforms specific assemblies. Than I just used an own solution for every platform to link anything together in an executable.

Overall in the end I didn't write that much code. It was mainly the process of finding the reasons why things were crashing or even worse just not working at all. I expect that I think I will still need some days to reintegrate everything to the current versions of CoreRT and the runtime and clean it up.

[jkotas]

As @MichalStrehovsky said. Any changes for the ARM64 port would be highly appreciated!

other runtime changes are platform specific and therefore I can not make them public

This can wait for later if there is more interest. We would need to have setup like what Mono has to deal with the licensing: https://www.mono-project.com/docs/about-mono/supported-platforms/xbox-one/ or https://www.mono-project.com/docs/about-mono/supported-platforms/playstation4/

[RalfKornmannEnvision]

FYI: I started to commit the ARM64 parts to https://github.com/RalfKornmannEnvision/corert/tree/ARM64

[RalfKornmannEnvision]

Seems like I cannot get it working anymore. I added the missing generation of CFI unwind data to RyuJit and that worked well. But many methods fail to compile with an error during register allocation. This might be caused by using a RyuJit that is compiled for X64 but should generate ARM64 code. I understand that this is not a supported scenario but it worked fine the last time. Or maybe because I am using the newest RyuJit version from GitHub while CoreRT needs an older version. Last time I needed to go back but I got an error message that told me that the newset one has an incompatible interface. This time at least the interfaces seems not to have changed.

[MichalStrehovsky]

It doesn't look like there has been a breaking change in the runtime repo recently. The tip of master should work. Note there was a RyuJIT change recently that broke the contract and didn't update the interface GUID in the past months, but if you use tip of both runtime and CoreRT, it should be fine.

I submitted #8242 to switch to the latest RyuJIT - let's see what the CI says to rule out a more general problem.

[RalfKornmannEnvision]

Using a compiled regular clrjit from the same runtime version works fine. I will try to build a Windows ARM64 version instead of the Unix-ARM64 one I was using so far. In the end I need the Unix version but at least I can check if it will show the same errors. 

[MichalStrehovsky]

The CI on the pull request to update RyuJIT is pretty red so this looks like a more general problem. I'll build a debug version of RyuJIT and see if I can spot the problem. Try rewinding the runtime repo to commit 98b6284e020845c04bc7b1cefdcd01ffe7a4a8c0 (this is the RyuJIT that is being used in CoreRT right now).

[RalfKornmannEnvision]

Thank's for the hint I will give it a try

[MichalStrehovsky]

The problem in the CI is likely a different problem. The JIT interface GUID didn't change, but runtime still changed how hardware intrinsics are reported and the tests that throw exceptions end up getting an infinite recursion because an intrinsic wasn't expanded by RyuJIT. It's likely unrelated to the register allocation problem you're seeing.

[RalfKornmannEnvision]

Well going back to the older version had at least some effect. There are now much less methods that fail to compile and it's an different assert. To be more precise it's now only one assert while there multiple once before. This issue seems to be somewhat related to System.Numerics.Vector`1 as all methods that are falling to compile have at least one argument of this type. The generic types are different.

Examples: 

Assertion failed '!regState->rsIsFloat' in 'System.SpanHelpers:LocateFirstFoundByte(System.Numerics.Vector`1[System.Byte]):int' during 'Linear scan register alloc' (IL size 73)

Assertion failed '!regState->rsIsFloat' in 'System.Numerics.Vector:BitwiseOr(System.Numerics.Vector1[System.UInt16],System.Numerics.Vector1[System.UInt16]):System.Numerics.Vector`1[System.UInt16]' during 'Linear scan register alloc' (IL size 13)

I will try to have a closer look. But to be honest I am not sure if I can find and fix this issue. Adding the missing CFI information is one thing but this seems to be much deeper inside the whole jit process. 

[MichalStrehovsky]

Where is that assertion? Could it be this one?

https://github.com/dotnet/runtime/blob/6072e4d3a7a2a1493f514cdf4be75a3d56580e84/src/coreclr/src/jit/regalloc.cpp#L177

I see this code path is in an if/else that appears to be dealing with HFA. It's quite likely we're not computing the HFA shape of Vector<T> here:

corert/src/ILCompiler.Compiler/src/Compiler/VectorOfTFieldLayoutAlgorithm.cs

Lines 65 to 68 in 9d8bd29

	public override ValueTypeShapeCharacteristics ComputeValueTypeShapeCharacteristics(DefType type)
	{
	return _fallbackAlgorithm.ComputeValueTypeShapeCharacteristics(type);
	}

It should probably be something along the lines of:

corert/src/ILCompiler.Compiler/src/Compiler/VectorFieldLayoutAlgorithm.cs

Lines 92 to 102 in 9d8bd29

	if (type.Context.Target.Architecture == TargetArchitecture.ARM64 &&
	type.Instantiation[0].IsPrimitiveNumeric)
	{
	return type.InstanceFieldSize.AsInt switch
	{
	8 => ValueTypeShapeCharacteristics.Vector64Aggregate,
	16 => ValueTypeShapeCharacteristics.Vector128Aggregate,
	_ => ValueTypeShapeCharacteristics.None
	};
	}
	return ValueTypeShapeCharacteristics.None;

[RalfKornmannEnvision]

Yes, it's this one. It's in line 178 in my version but it's the same code.

Thank's for your help as I would properly never found the connection between this assert and the code part in the IL Compiler. I will have a look if I can figure out what's wrong and see if I can fix it

[MichalStrehovsky]

I think you can just copy-paste the implementation from VectorFieldLayoutAlgorithm.cs - it should do the right thing. Fingers crossed it helps, but it really feels like it should help taking the if (argDsc->lvIsHfaRegArg()) path in RyuJIT instead of the asserting one.

[RalfKornmannEnvision]

Thank you very much. Now it runs without any asserts to the point were it needs to create the object writer. This was the next thing on my list were I need to reintegrate my old changes.