Will Integrant support parallel initialization?

[domkm]

For example, component A depends on B and C but both B and C are independent and can take quite some time to initialize. Could Integrant start B and C concurrently instead of serially?

[weavejester]

In theory that possible, but it would take a bit of work to achieve. Right now the components are sorted and started serially; we'd need a more sophisticated system to figure out which could be started in parallel.

[hmaurer]

@weavejester how sophisticated would this be? I assume you would need to model the dependency as a directed acyclic graph, then traverse it, starting from nodes with no incoming edges, and at each step loading all selected nodes in parallel? Or am I missing something?

[danielcompton]

You'd also need to handle threading, Integrant might need to provide a thread pool, or let the user provide one. It seems possible, but would need some thought to keep it easy for the user.

[weavejester]

The dependencies are already modelled as a graph, which can be retrieved with the dependency-graph function. However, there are a few difficulties around initiating concurrently:

1. The order of components starting is no longer deterministic
2. We need to build our own iteration mechanism
3. We need to keep a list of all dependencies that have begun to be initiated, as we can run into the same dependency multiple times even if the graph is acyclic
4. We need to halt all threads on an exception, and return the exception
5. We might need some manner of thread pool
6. Certain keys will take a very short time to initiate; these may actually be slower with the thread overhead
7. Maybe other things I haven't thought of

[deoqc]

From Erlang Programming Rules:

3.12 Make code as deterministic as possible

For example, suppose one process has to start five parallel processes and then check that they have started correctly, suppose further that the order in which these five are started does not matter.

We could then choose to either start all five in parallel and then check that they have all started correctly but it would be better to start them one at a time and check that each one has started correctly before starting the next one.

Well, probably the default should be start serially.

Only if is really needed, it could explicitly be asked to start something in parallel.

[serioga]

My solution for this problem https://gist.github.com/serioga/1c2f5a6ef771a267a6e726a066e2bf39

[serioga]

Parallel initialization addon for integrant https://github.com/serioga/webapp-clojure-2020/blob/master/src/lib/integrant/async.clj