Make improvements to signal handling on .NET applications

[davidfowl]

Background and Motivation

Many systems use SIGTERM as a way to signal a graceful shutdown. To make this work well in container scenarios, we need to start the shutdown process and wait for some timeframe until it's done. To coordinate that, Microsoft.Extensions.Hosting sets up a bunch of events to allow the main thread to exit before continuing the process exit handler. Here's what that would look like (without the dependencies):

using System;
using System.Threading;

var waitForProcessShutdownStart = new ManualResetEventSlim();
var waitForMainExit = new ManualResetEventSlim();

AppDomain.CurrentDomain.ProcessExit += (sender, e) =>
{
    // We got a SIGTERM, signal that graceful shutdown has started
    waitForProcessShutdownStart.Set();

    Console.WriteLine("Waiting for main");
    // Don't unwind until main exists
    waitForMainExit.Wait();
};

Console.WriteLine("Waiting for shutdown");
// Wait for shutdown to start
waitForProcessShutdownStart.Wait();

// This is where the application performs graceful shutdown


// Now we're done with main, tell the shutdown handler
waitForMainExit.Set();

The above code shows how a user could set this up today. It's much more complex than the windows equivalent (handling Ctrl+C) for a couple of reasons:

• SIGTERM is the signal used to communicate starting a graceful shutdown, this means we need to stop the app from shutting down until some point.
• We want the main thread to continue executing for as long as it can until it has unwound. This lets us run any clean up code or logging before exit (see IHost.RunAsync() never completes #44086)
• It can result in deadlocks if someone calls Environment.Exit at the wrong point in the application.

This is what waiting for CTRL+C looks like:

var waitForProcessShutdownStart = new ManualResetEventSlim();
        
Console.CancelKeyPress += (sender, e) =>
{
    e.Cancel = true;

    waitForProcessShutdownStart.Set();
};

Console.WriteLine("Waiting for shutdown");

// Wait for shutdown to start
waitForProcessShutdownStart.Wait();

// Do graceful shutdown here

The runtime itself handles e.Cancel = true and doesn't shut down the process after the event handler runs. Instead the application can use this to coordinate letting the application gracefully unwind from the main thread.

The request here is to treat SIGTERM like Ctrl+C and support e.Cancel.

cc @janvorli @stephentoub @kouvel

Proposed API

namespace System.Runtime.InteropServices
{
+    public enum Signal // This may need more unique name
+    {
+        SIGHUP = 1,
+        SIGINT = 2,
+        SIGQUIT = 3,
+        SIGTERM = 15,
+    }

+    public class SignalContext
+    {
+        public Signal Signal { get; }
+        public bool Cancel { get; set; }
+    }

+    public struct SignalHandlerRegistration : IDisposable
+    {
+        public static SignalHandlerRegistration Create(Signal signal, Action<SignalContext> handler);
+    }
}

NOTES:

• SignalContext.Signal provides the signals that fired to cause the event to trigger so that can be checked in the callback to take action if the same handler was used for different registrations.
• SignalContext.Cancel cancels default processing.

We will map windows behavior to linux signal names:

• CTRL_C_EVENT = SIGINT
• CTRL_BREAK_EVENT = SIGINT
• CTRL_SHUTDOWN_EVENT - SIGTERM
• CTRL_CLOSE_EVENT - SIGTERM
• CTRL_LOGOFF_EVENT - Not mapped

Usage Examples

using var reg = SignalHandlerRegistration.Register(Signal.SIGINT, context => 
{
     context.Cancel = true;

     // Start graceful shutdown
});

Waiting synchronously on shutdown to start:

var waitForProcessShutdownStart = new ManualResetEventSlim();
        
using var reg = SignalHandlerRegistration.Register(Signal.SIGINT, context => 
{
    context.Cancel = true;

    waitForProcessShutdownStart.Set();
});

Console.WriteLine("Waiting for shutdown");

// Wait for shutdown to start
waitForProcessShutdownStart.Wait();

The hosting model in Microsoft.Extensions.Hosting will look like this in .NET 6:

public class ConsoleLifetime : IHostLifetime, IDisposable
{
    ...

    private IHostApplicationLifetime ApplicationLifetime { get; }

    private SignalHandlerRegistration _sigterm;
    private SignalHandlerRegistration _sigInt;

    public Task WaitForStartAsync(CancellationToken cancellationToken)
    {
        Action<SignalContext> handler = OnShutdownSignal;

        _sigInt = SignalHandlerRegistration.Register(Signal.SIGINT, handler);
        _sigterm = SignalHandlerRegistration.Register(Signal.SIGTERM, handler);

        // Console applications start immediately.
        return Task.CompletedTask;
    }

    private void OnShutdownSignal(SignalContext context)
    {
        context.Cancel = true;

        ApplicationLifetime.StopApplication();
    }

    public void Dispose()
    {
        _sigterm.Dispose();
        _sigInt.Dispose();
    }

    ...
}

Alternative Designs

Don't add new API but overload the existing Console.CancelKeyPress. This would cover one specific scenario but wouldn't handle the arbitrary signals.

Risks

None

[jkotas]

Change that introduced the current behavior: dotnet/coreclr#4309

I am not sure whether it is a good idea to overload the existing ProcessExit event to do this. There is a lot of code out there that uses this event for last minute cleanup during shutdown. It is not very compatible with everything running gracefully for a while until the actual shutdown happens.

[davidfowl]

I'm fine with introducing a new event if that's what it takes, I just want the problem to be solved by the runtime itself instead of the libraries (which does a poor job at it).

[davidfowl]

OK let's design an API. Today we have AppDomain.ProcessExit, this is the one that fires when it's too late. We also have Console.CancelKeyPress that means Ctrl+C. I'd like an API that handles CTRL+C or SIGINT or SIGTERM, that triggers an event, but does not end the process.

namespace System
{
    public static class Environment
    {
+        public static Task WaitForShutdownSignalAsync();
+        public static void WaitForShutdownSignal();
    }
}

Usage

using System;

// Blocking wait for shutdown to start
Environment.WaitForShutdownSignal();

// This is where the application performs graceful shutdown

Non-blocking wait

using System;

// Non-Blocking wait for shutdown to start
await Environment.WaitForShutdownSignalAsync();

// This is where the application performs graceful shutdown

I know this has the potential to get into the generic signal handling discussion but I'd like us to focus on the main requirements:

• Applications want to write the same code on across all platforms to wait on a shutdown signal
• These signals DO NOT shut the application down. They are just signals that give the application a chance to unwind and cleanup.

[jkotas]

• It should not be on Process. Process is for working with other processes, not your own process. I would put it on Environment.
• Does it need to take argument that describes the kind of signals you want to wait for and/or the kind of signal that was received? Does it need to give you an option to resume the default processing of the signal? Ideally, this API would be capable enough to allow the existing Console Control+C handler reimplemented by wrapping this API.

[davidfowl]

It should not be on Process. Process is for working with other processes, not your own process. I would put it on Environment.

Updated.

Does it need to take argument that describes the kind of signals you want to wait for and/or the kind of signal that was received?

I wanted to avoid that for this change. We can always add overloads that take an enum or flags enum with the signals to care about.

Does it need to give you an option to resume the default processing of the signal?

It could. And like today it would be confusing as there could be multiple waiters an each who can decide what the global state of resuming the signal handling should be. I think not having a callback API means it won't matter much in practice since my entire continuation will execute before signal processing resumes anyways.

Ideally, this API would be capable enough to allow the existing Console Control+C handler reimplemented by wrapping this API.

I don't think this API should be a replacement for the current ConsoleKeyProcess handler. I don't think it should initiate shutdown at all, just leave that up to the application.

[stephentoub]

I wanted to avoid that for this change. We can always add overloads that take an enum or flags enum with the signals to care about.

Why? What's special about these specific signals other than they're the ones required today? If we're going to add a general signal listening mechanism, it should be generalized.

That said, what is the expected behavior on Windows?

I don't think this API should be a replacement for the current ConsoleKeyProcess handler. I don't think it should initiate shutdown at all, just leave that up to the application.

The console handler doesn't initiate shutdown. It let's you intercept the signal that otherwise shuts down. It only has to do something special in this regard if you choose not to intercept.

I think not having a callback API means it won't matter much in practice since my entire continuation will execute before signal processing resumes anyways.

Not if it's async. Also, you can't run arbitrary managed code in signal handiers, so the callback needs to be made async from the signal handler anyway. For reference, you can see this in how we handle ctrl-C in Console. When a signal arrives, our registered signal handler queues our handling of it and then also immediately invokes the next handler in the chain:

runtime/src/libraries/Native/Unix/System.Native/pal_signal.c

Lines 43 to 68 in 4523fbf

	static void SignalHandler(int sig, siginfo_t* siginfo, void* context)
	{
	// Signal handler for signals where we want our background thread to do the real processing.
	// It simply writes the signal code to a pipe that's read by the thread.
	uint8_t signalCodeByte = (uint8_t)sig;
	ssize_t writtenBytes;
	while ((writtenBytes = write(g_signalPipe[1], &signalCodeByte, 1)) < 0 && errno == EINTR);
	if (writtenBytes != 1)
	{
	abort(); // fatal error
	}
	// Delegate to any saved handler we may have
	// We assume the original SIGCHLD handler will not reap our children.
	if (sig == SIGCONT || sig == SIGCHLD || sig == SIGWINCH)
	{
	struct sigaction* origHandler = OrigActionFor(sig);
	if (origHandler->sa_sigaction != NULL &&
	(void*)origHandler->sa_sigaction != (void*)SIG_DFL &&
	(void*)origHandler->sa_sigaction != (void*)SIG_IGN)
	{
	origHandler->sa_sigaction(sig, siginfo, context);
	}
	}
	}

There's then a thread processing this queue, which handles the SIGINT by invoking the user's callback:

runtime/src/libraries/Native/Unix/System.Native/pal_signal.c

Lines 109 to 117 in 79ae74f

	if (signalCode == SIGQUIT || signalCode == SIGINT)
	{
	// We're now handling SIGQUIT and SIGINT. Invoke the callback, if we have one.
	CtrlCallback callback = g_ctrlCallback;
	CtrlCode ctrlCode = signalCode == SIGQUIT ? Break : Interrupt;
	if (callback != NULL)
	{
	callback(ctrlCode);
	}

And if the callback says the ctrl-C shouldn't be canceled, at that point we can't still rely on the default tear-down-the-app-behavior, since the original signal is long gone, so we clear out our handler and re-send the signal, such that this time it'll tear things down:

runtime/src/libraries/Native/Unix/System.Native/pal_signal.c

Lines 191 to 199 in 79ae74f

	void SystemNative_RestoreAndHandleCtrl(CtrlCode ctrlCode)
	{
	int signalCode = ctrlCode == Break ? SIGQUIT : SIGINT;
	#ifdef HAS_CONSOLE_SIGNALS
	UninitializeTerminal();
	#endif
	sigaction(signalCode, OrigActionFor(signalCode), NULL);
	kill(getpid(), signalCode);
	}

[davidfowl]

Why? What's special about these specific signals other than they're the ones required today? If we're going to add a general signal listening mechanism, it should be generalized.

My motivation is to fix bugs we cannot work around today to do with graceful shutdown. What makes these signals special is what I said here:

"Many systems use SIGTERM as a way to signal a graceful shutdown."

The pattern that repeats itself in many of these systems (Kubernetes, Docker, Systemd), send a SIGTERM to give the application some time to shutdown, then follow up with a SIGKILL, if the app hasn't shutdown in some timeframe. SIGINT, is the dev scenario where the developer hits CTRL+C on the command line.

That said, what is the expected behavior on Windows?

On windows, this usually ends up being CTRL+C.

Last time this conversation stalled at Mono.Posix. I don't have a problem if you think we should make it more general purpose, but these signals are special because of the shutdown semantics other systems are applying to these signals.

Not if it's async. Also, you can't run arbitrary managed code in signal handiers, so the callback needs to be made async from the signal handler anyway. For reference, you can see this in how we handle ctrl-C in Console. When a signal arrives, our registered signal handler queues our handling of it and then also immediately invokes the next handler in the chain:

Right, I figured this would be the case. (though I didn't look at the code). I don't have a strong opinion on if we should allow this. Again my motivation is about handling these buggy shutdown cases that are hard to workaround today without pinvoking.