Limits on job mix

[svatosFZU]

Hi,
I would like to ask if it is possible to put in a limit on one job kind. Until now, I was running only one kind of jobs (i.e. all jobs have the same requirements on CPU and memory). Now, I would like to try to send a mix of jobs. For example, to 128-core nodes of Karolina, I would like to send mix of 32-core jobs and single-core jobs. Leaving the HQ to just mix them without constraints will not work as too many of those single-core jobs could kill the worker node. So, is there a way to tell the HQ to mix the jobs to fill the node in such a way that there is only configurable number (32 or 64 would make sense in this case) of those single-core jobs?

[spirali]

You can create an artificial resource that will limit the single core tasks.

Example:

Create resources on worker (32 artifical resources "foo")

hq worker start --resource "foo=sum(32)"

If you submit task like:

hq submit --resource foo=1 -- sleep 1

Then it will need 1 cpu and 1 foo. So it will not run more than 32 of these tasks on the worker because it has only 32 foo resources.

[svatosFZU]

Thanks. Now, just be sure that I understand it correctly, lets be specific. The worker properties are set by allocation queue. So, I would start it with

hq alloc add slurm --resource "score=sum(32)" ...

and then I would send the single-core jobs with

hq submit --resource score=1 --cpus=1

and the 32-core jobs with

hq submit --cpus=32 ...

[spirali]

Yes, this is the right usage.

Btw: If your 1 cpu tasks are limited by the memory, you can also use:

hq submit --resource mem=100 # This task needs 100MB of RAM.

And it will not start the task if the sum of the resource "mem" of all running tasks would exceed the RAM. So you do not need an artificial resource, because "mem" resource is detected automatically. But artificial resources are nothing bad. It is up to you.

[svatosFZU]

Thanks, that is what I needed to know.
closing

[svatosFZU]

Hi,
I would have one follow-up question related to this. I am testing it on Barbora (36-core WNs). The situation is simpler there as I do not need mixed jobs, i.e. I need to submit either one 36-core job or 36 single-core jobs. But I have difficulties to do the latter one. It seems that the HyperQueue is trying to submit the jobs even though it does not have enough to fill the WN. As result, I have HQ workers which ran only few single-core jobs. That is quite inefficient. Is there a way for me to tell the HyperQueue to submit allocation only if it has enough work to fill it completely?

[Kobzol]

Could you please clarify what is WN? Just so that I don't miss anything important :)

The autoallocator is currently quite eager and starts allocations as soon as there is something to be computed. "Filling an allocation with work" is a very non-trivial property, it's not really possible to know beforehand if a given set of tasks will finish very quickly, or if they will run for a long time, and thus how many allocations they will need. In fact if the user doesn't specify it, we don't even know how many cores do nodes have in a given allocation queue.

I guess that we could add some different submit mode or a parameter that would say something like "only start an allocation once you have N tasks buffered".

[spirali]

Could you please clarify what is WN? Just so that I don't miss anything important :)

The autoallocator is currently quite eager and starts allocations as soon as there is something to be computed. "Filling an allocation with work" is a very non-trivial property, it's not really possible to know beforehand if a given set of tasks will finish very quickly, or if they will run for a long time, and thus how many allocations they will need. In fact if the user doesn't specify it, we don't even know how many cores do nodes have in a given allocation queue.

I guess that we could add some different submit mode or a parameter that would say something like "only start an allocation once you have N tasks buffered".

Technically, I can enrich NewWorkerAllocationResponse with some more information how would be new workers utilized (e.g. as some ratio). This can allow to expose to the user some configuration like --min-utilization for queue. If it is set to for example to --min-utilization=0.5, it would not start a new allocation when the worker is not utilized at least by 50% (= at least one resource (e.g. cpus) is not utilized at least by 50%). Scheduler is already computing some of these numbers for internal usage. (Of course it assumes that we already have some queue resource discovery mechanism). But I am not sure that it is what we are asked for.

[svatosFZU]

By WN, I mean worker node, i.e. the available hardware.
For the filling it with work through its lifetime, I understand it is non-trivial regarding the various payload length. But I am talking about just the beginning of the job. I know how big the machines are, I can tell the HyperQueue beforehand, how many cores there are to fill. To summarize, the functionality I am asking about is to tell the HyperQueue: do not submit a batch job unless there is enough jobs to use all cpus (and maybe also memory) of the node. The min-utilization could be a way to go.

[svatosFZU]

Hi,
could I ask if there is any estimate for when the --min-utilization could become available?

[spirali]

I am going to update the scheduler query next week, then it has to be integrated into autoalloc that is a question on @Kobzol.

[Kobzol]

Autoalloc will need some changes before that, and there are some refactorings blocking that, but I think that it should be doable in 1-2 months.