Specifying processor usage

[Jes-s-L]

Hi!

I am currently using the default 'multithreading' over all cores, however I would like to be able to restrict the number of processors used. Additionally, I am calling some additional Julia packages for use in a TemplateExpressionSpec, including ONNXRunTime for a neural network using:

from pysr import PySRRegressor, TemplateExpressionSpec, jl

jl.seval("""
Pkg.add("ONNXRunTime")
import ONNXRunTime as ORT

model = ORT.load_inference( ... )
...
""")

I noticed specifying procs is ignored when using multithreading, and when using multiprocessing I am unsure how to correctly import Julia packages or specify 'global' variables/models as it seems my current method does not work. What would be the recommended method for this workflow?

Thank you for the help!

[MilesCranmer]

I would recommend trying to stick with multithreading unless there's a specific reason to switch. Julia doesn't have a GIL so the multithreading is actually parallel.

Btw, can you share the error you are seeing?

In terms of packages, there is a worker_imports argument for importing packages on workers.

[MilesCranmer]

@MilesCranmerBot can you please backport the worker_imports feature to v1 in a new PR?

[Jes-s-L]

I had a go at using worker_imports with multiprocessing in version 2.0.0a1, but it seems the script hung at the 'testing pipeline' step. Unfortunately, not entirely sure what happened, there was no obvious error.

I assume it would be best for now to stick with multithreading, and just a sanity check - is multithreading on a subset of the total number of cores possible?

[MilesCranmer]

The default number of threads is "auto" which lets Julia decide. You can also force the number of threads by restarting Python with an environment variable PYTHON_JULIACALL_THREADS=8, for example.

[Jes-s-L]

Setting PYTHON_JULIACALL_THREADS and procs does reduce the CPU usage, however it seems to run on many more cores than specified but only at about 40% utilisation. However, the initial usage to compile Julia seems to be correctly limited to the specified number of cores.

Apologies I'm not very familiar with the technicalities of parallelising. Would it be caused by something in Python instead?

[MilesCranmer]

however it seems to run on many more cores than specified but only at about 40% utilisation

It' normal to have # threads > # of cores to address this type of thing. If the load isn't maxed out then I would just increase the # of threads and see if that helps