Error on windows: daemonic processes are not allowed to have children

[M-Kristof]

I am encountering a problem when running lists of learners in a jupyterlab notebook on a windows machine.

For example, when trying out 'Balancing learner' of the introductory notebook,

import adaptive
adaptive.notebook_extension()

# Import modules that are used in multiple cells
import holoviews as hv
import numpy as np
from functools import partial
import random

def h(x, offset=0):
    a = 0.01
    return x + a**2 / (a**2 + (x - offset)**2)

learners = [adaptive.Learner1D(partial(h, offset=random.uniform(-1, 1)),
            bounds=(-1, 1)) for i in range(10)]

bal_learner = adaptive.BalancingLearner(learners)
runner = adaptive.Runner(bal_learner, goal=lambda l: l.loss() < 0.01)
runner.live_info()

I get the following error message:

future: <Task finished coro=<_wrap_awaitable() done, defined at C:\ProgramData\Anaconda3\lib\asyncio\tasks.py:596> exception=AssertionError('daemonic processes are not allowed to have children')>
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\lib\asyncio\tasks.py", line 603, in _wrap_awaitable
    return (yield from awaitable.__await__())
  File "C:\ProgramData\Anaconda3\lib\site-packages\distributed\nanny.py", line 245, in start
    response = await self.instantiate()
  File "C:\ProgramData\Anaconda3\lib\site-packages\distributed\nanny.py", line 328, in instantiate
    result = await self.process.start()
  File "C:\ProgramData\Anaconda3\lib\site-packages\distributed\nanny.py", line 502, in start
    await self.process.start()
  File "C:\ProgramData\Anaconda3\lib\site-packages\distributed\process.py", line 33, in _call_and_set_future
    res = func(*args, **kwargs)
  File "C:\ProgramData\Anaconda3\lib\site-packages\distributed\process.py", line 190, in _start
    process.start()
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\process.py", line 110, in start
    'daemonic processes are not allowed to have children'

It works fine on linux.

[akhmerov]

Seems to be a general limitation of multiprocessing on windows: https://stackoverflow.com/questions/6974695/python-process-pool-non-daemonic

[basnijholt]

This shouldn't happen because Windows should use distributed instead of multiprocessing:

adaptive/adaptive/runner.py

Lines 43 to 60 in 5db0f73

	if os.name == "nt":
	if with_distributed:
	_default_executor = distributed.Client
	_default_executor_kwargs = {"address": distributed.LocalCluster()}
	else:
	_windows_executor_msg = (
	"The default executor on Windows for 'adaptive.Runner' cannot "
	"be used because the package 'distributed' is not installed. "
	"Either install 'distributed' or explicitly specify an executor "
	"when using 'adaptive.Runner'."
	)
	_default_executor_kwargs = {}
	def _default_executor(*args, **kwargs):
	raise RuntimeError(_windows_executor_msg)
	warnings.warn(_windows_executor_msg)

Can you check if os.name == "nt" and adaptive.runner. _default_executor?

[M-Kristof]

The answer is yes to both. It is using distributed.

[jbweston]

This may well be a bug in distributed; further investigation needed.

[jbweston]

possibly related: dask/distributed#2142 dask/distributed#2718

[basnijholt]

We were able to reproduce this in #247 where we added tests for Windows.

It seems like you should just be able to use a ProcessPoolExecutor. When that PR is merged, it will become the default.

[jbweston]

@M-Kristof, seems that #247 was merged and tests are passing on Windows.

I'll close this issue and you can re-open it if the problem resurfaces.