Hit inotify user limit on linux when you start and stop hosts created with Host.CreateDefaultBuilder(args) in a tight loop

[foriequal0]

Description

Starting and Stopping a generic host created with Host.CreateDefaultBuilder(args) in a tight loop can cause inotify limit exception.

Reproduction Steps

using System.Threading.Tasks;
using Microsoft.Extensions.Hosting;

namespace App
{
    public class Program
    {
        public static async Task Main(string[] args)
        {
            for (var i = 0; i < 100; i++)
            {
                using var host = await Host.CreateDefaultBuilder(args).StartAsync();
                await host.StopAsync();
            }
        }
    }
}

Expected behavior

JsonConfigurationProvider is disposed when the host is stopped and disposed.

Actual behavior

Unhandled exception. System.IO.IOException: The configured user limit (128) on the number of inotify instances has been reached, or the per-process limit on the number of open file descriptors has been reached.
   at System.IO.FileSystemWatcher.StartRaisingEvents()
   at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed()
   at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean value)
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter)
   at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(String filter)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__1_0()
   at Microsoft.Extensions.Primitives.ChangeToken.OnChange(Func`1 changeTokenProducer, Action changeTokenConsumer)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(FileConfigurationSource source)
   at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(IConfigurationBuilder builder)
   at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
   at Microsoft.Extensions.Hosting.HostBuilder.BuildAppConfiguration()
   at Microsoft.Extensions.Hosting.HostBuilder.Build()
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostBuilderExtensions.StartAsync(IHostBuilder hostBuilder, CancellationToken cancellationToken)
   at WebApplication1.Program.Main(String[] args) in /home/foriequal0/workspace/nk-backend/Library/WebApplication1/Program.cs:line 12
   at WebApplication1.Program.<Main>(String[] args)

Process finished with exit code 134.

Regression?

No response

Known Workarounds

Dispose manually file providers

                using var host = await Host.CreateDefaultBuilder(args).StartAsync();
                await host.StopAsync();

                var configuration = host.Services.GetService<IConfiguration>() as ConfigurationRoot;
                foreach (var provider in configuration?.Providers?.OfType<FileConfigurationProvider>() ?? Array.Empty<FileConfigurationProvider>())
                {
                    (provider.Source.FileProvider as IDisposable)?.Dispose();
                }

Configuration

Ubuntu 21.10, .NET 6

Other information

No response

Tagging subscribers to this area: @dotnet/area-extensions-hosting
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

Starting and Stopping a generic host created with Host.CreateDefaultBuilder(args) in a tight loop can cause inotify limit exception.
I found that JsonConfigurationProvider is never disposed.

Reproduction Steps

using System.Threading.Tasks;
using Microsoft.Extensions.Hosting;

namespace App
{
    public class Program
    {
        public static async Task Main(string[] args)
        {
            for (var i = 0; i < 100; i++)
            {
                using var host = await Host.CreateDefaultBuilder(args).StartAsync();
                await host.StopAsync();
            }
        }
    }
}

Expected behavior

JsonConfigurationProvider is disposed when the host is stopped and disposed.

Actual behavior

Unhandled exception. System.IO.IOException: The configured user limit (128) on the number of inotify instances has been reached, or the per-process limit on the number of open file descriptors has been reached.
   at System.IO.FileSystemWatcher.StartRaisingEvents()
   at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed()
   at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean value)
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter)
   at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(String filter)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__1_0()
   at Microsoft.Extensions.Primitives.ChangeToken.OnChange(Func`1 changeTokenProducer, Action changeTokenConsumer)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(FileConfigurationSource source)
   at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(IConfigurationBuilder builder)
   at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
   at Microsoft.Extensions.Hosting.HostBuilder.BuildAppConfiguration()
   at Microsoft.Extensions.Hosting.HostBuilder.Build()
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostBuilderExtensions.StartAsync(IHostBuilder hostBuilder, CancellationToken cancellationToken)
   at WebApplication1.Program.Main(String[] args) in /home/foriequal0/workspace/nk-backend/Library/WebApplication1/Program.cs:line 12
   at WebApplication1.Program.<Main>(String[] args)

Process finished with exit code 134.

Regression?

No response

Known Workarounds

No response

Configuration

Ubuntu 21.10, .NET 6

Other information

No response

Author:	foriequal0
Assignees:	-
Labels:	untriaged, area-Extensions-Hosting
Milestone:	-

[foriequal0]

I found that FileConfigurationSource doesn't implement IDisposable who holds PhysicalFileProvider.
I think it should implement IDisposable, and FileConfigurationProvider should dispose it too.

[foriequal0]

There were already an exact issue for this, and it was closed: #37610

I saw the comment from @mjrousos.
FileConfigurationProvider already implements IDisposable, and PhysicalFileProvider does too.
I don't see why implementing IDisposable for FileConfigurationSource would not be viable.

[srollinet]

I have the same issue and I try to debug it to see what happens, but I don't understand what happens

I profiled the application using your original code

using System.Threading.Tasks;
using Microsoft.Extensions.Hosting;

namespace App
{
    public class Program
    {
        public static async Task Main(string[] args)
        {
            for (var i = 0; i < 100; i++)
            {
                using var host = await Host.CreateDefaultBuilder(args).StartAsync();
                await host.StopAsync();
            }
        }
    }
}

Looking at the calls number, the PhysicalFileProvider is disposed, but its internal PhysicalFilesWatcher is never disposed

I tried to debug the code step by step using Rider and this is also what I observe. When I step into this method, the debugger tells me that _fileWatcher is null, even when it has been created.

runtime/src/libraries/Microsoft.Extensions.FileProviders.Physical/src/PhysicalFileProvider.cs

Lines 208 to 218 in be98e88

	protected virtual void Dispose(bool disposing)
	{
	if (!_disposed)
	{
	if (disposing)
	{
	_fileWatcher?.Dispose();
	}
	_disposed = true;
	}
	}

I suspect the debugger / profiler is lying to me somehow. Otherwise something really strange is happening.