BlockingIOError: [Errno 11] Resource temporarily unavailable #176
Description
We run this on multiple webtop containers (https://github.com/linuxserver/docker-webtop).
The containers can sometimes become IO constrained due to heavy IO. This causes systemctl3.py to exit with the following error:
sys.exit(run(command, *modules))
File "/usr/local/bin/systemctl", line 6502, in run
exitcode = is_not_ok(systemctl.default_system())
File "/usr/local/bin/systemctl", line 5722, in default_system
return self.start_system_default(init = init)
File "/usr/local/bin/systemctl", line 5732, in start_system_default
sig = self.init_loop_until_stop(services)
File "/usr/local/bin/systemctl", line 6087, in init_loop_until_stop
self.read_log_files(units)
File "/usr/local/bin/systemctl", line 5905, in read_log_files
os.write(1, content)
BlockingIOError: [Errno 11] Resource temporarily unavailable
Once it exits, it's child processes are moved to PID 1 (we use s6 as the PID 1 and run systemctl as an s6 service to handle systemd services for convenience).
systemctl3.py then enters an infinite loop of trying to start the services which are already running. They exit immediately due to PID file lock contention, and then systemctl3.py tries to start them again.
This infinite loop uses 100% CPU:
753301 root 20 0 259484 97988 7180 R 99.7 0.0 4:58.09 systemctl
Is there any way we can remove the IO timeout? Having systemctl hang is much more preferable to an exit, as the contention will eventually subside and resources will become available again.