default worker priorities don't play nicely with smaller worker count

[bgentry]

The --worker-count arg defaults to 6. However, when trying to migrate over to this new version I decided to try it with 1 worker locally while I worked through bugs in my jobs.

I noticed that I had a bunch of jobs piling up locally, and it took me awhile to realize it was because of the --worker-priorities option:

--worker-priorities [LIST]   List of priorities to assign to workers, unspecified workers take jobs of any priority (default: 10,30,50)

Because that defaults to 10,30,50, when running fewer than 4 workers, no jobs with a priority number above 50 (such as the default 100) will run. It seems like these options will probably need to be set in tandem for many people.

Maybe there's a better way of setting these options up out-of-the-box to avoid confusing behavior like this?

[chanks]

Ah, yeah, this was discussed this a bit in an earlier issue (#213 (comment)) but I neglected to actually put it on my to-do list 😳 Thanks for opening this. I can look into it tomorrow.

[bgentry]

@chanks cool. Awesome work on v1 btw! The upgrade was pretty painless, mainly had to go through and fix jobs to account for the switch from arg string keys to symbol keys. I'm not taxing it very much but it seems to be flying along now 🎉

[chanks]

Just pushed some commits to master to fix this. The default priorities are still [10, 30, 50, nil, nil, nil], but when the worker count is changed, the workers take their priorities from the end of the array rather than the front. So --worker-count=1 will give a single worker with priority nil (so it can work any job). --worker-count=4 will give four workers with priorities [50, nil, nil, nil], etc.

I also made the options for passing priorities more robust and safe. If the worker count exceeds the number of passed priorities the excess workers were always nil, but now you can be explicit by passing --worker-count=4 --worker-priorities=10,20,nil,nil, for example.

Try this out and let me know what you think. I sort of want to get rid of the --worker-count option and just use --worker-priorities instead, but I also kind of like the idea of having a simple --worker-count knob that's easier to tweak without needing to understand too much?

I still need to modify the buffer-size options as a result of the worker count, but I'm not sure yet what good algorithms for those would be...

[bgentry]

That sounds like a nice improvement. I wonder if instead of popping off the end of the priority list, you just changed the default priority list? Something like nil,50,30,10 or nil,10,30,50 would make sure all jobs still get worked, and might be less confusing than the scenario where you specify fewer workers than priorities and it starts at the end of the list.

[chanks]

I just pushed another few commits, so that setting --worker-count just sets the worker pool to that number of priority-ignorant workers. This is closer to the expected behavior of that setting (and the way it was in 0.x), and in particular guarantees the correct behavior of --worker-count=1. It also lets people get up and running without worrying about priority levels if they don't want to.

Also, the CLI now accepts any rather than nil in the priority list, which is clearer - the fact that those workers have a nil priority is an implementation detail and not something that needs to be exposed in the API.

So, the default behavior is --worker-priorities=10,30,50,any,any,any, and --worker-count=1 is equivalent to --worker-priorities=any.

[chanks]

Just released 1.0.0.beta3 with the above changes. Let me know if anything comes up!