Emit safepoints at function entry

[vchuravy]

Otherwise non-allocating functions like:

fib(x) = x <= 1 ? 1 : fib(x-1) + fib(x-2) will block GC from make progress.

This now inserts a safepoint at the beginning of every function.

Note:
Initially I wanted to insert these safepoints late and also into back-branches. That is currently not legal since at a safepoint we must gurantuee that all objects are rooted and legal. Legal means that the typetag is set correctly, and thus we shall not sink a store of a typetag past a safepoint, and rooting requires that stores of an object to a field shall not be sunk past a safepoint.
We can't currently gurantuee these properties after we already started optimizing the code.

@vtjnash had the idea that we could insert safepoints at the exit blocks of outer loops, that might be an idea for the future.

Besides inserting a safepoint here this has the additional overhead of requiring a lookup from a thread local variable + a load from the current ptls. In c76459c I hoisted the signal_page pointer load outside the fence.

@maleadt I made the emission a codegen feature so that we can turn it off for GPU code.

[vchuravy]

Ah shoot, LateLowerGC of course doesn't see these as safepoints. So yeah we need to introduce julia.safepoint as an opaque function and lower the safepoint in FinalLower

[JeffBezanson]

Very curious to see the performance impact of this. The function entry one we can maybe just go ahead with, but for loops I imagine we'll have to put state transitions around the loop instead of safepoints on every backedge.

[vtjnash]

I believe, for correctness, this must be inserted by codegen, before optimization passes run, as it adds a global side-effect (thread synchronization points) to the code. We can then write optimization passes that merge and delete them, or that convert them into safe-regions. But I am not sure it is legal to add them later.

[vchuravy]

as it adds a global side-effect (thread synchronization points) to the code.

The safepoint has a pair off:

     B.CreateFence(AtomicOrdering::SequentiallyConsistent, SyncScope::SingleThread);

Which according to https://llvm.org/docs/LangRef.html#syncscope is:

If an atomic operation is marked syncscope("singlethread"), it only synchronizes with and only participates in the seq_cst total orderings of other operations running in the same thread (for example, in signal handlers).

So from a memory model point it should be legal to insert them late?

The function entry one we can maybe just go ahead with, but for loops I imagine we'll have to put state transitions around the loop instead of safepoints on every backedge.

Yeah that's my feeling as well. Hopefully have time to work on this next week.

[vchuravy]

Jameson convinced me out-of-band that we can't late insert safepoints since LLVM might sink the tag store past a safepoint with out the memory fence. So this is doing the more minimal change of emitting a safepoint at the start of the function.

[vchuravy]

@nanosoldier runbenchmarks(ALL, vs=":master")

[vchuravy]

@nanosoldier runtests(ALL, vs = ":master")

[nanosoldier]

Your package evaluation job has completed - possible new issues were detected. A full report can be found here.

[nanosoldier]

Your benchmark job has completed - possible performance regressions were detected. A full report can be found here. cc @christopher-dG

[vtjnash]

Need to teach it to remove this safepoint after codgen for functions which don't interact with the runtime anyways (hack for @threadcall), and rebase

[KristofferC]

I've reverted this on 1.9 due to #47826 to get it off the milestone.