Listener won't enqueue jobs when buffer size is 0

[jasoncodes]

I’ve been testing --minimum-buffer-size=0 --maximum-buffer-size=0 in order to not lock jobs when there aren't enough free workers to run them immediately. The poller had an issue with this which was reported in #235 and fixed by 1736d3a and the same issue seems to exist with the listen code path.

Jobs run with a buffer size set to 0 thanks to the poller picking them up but they don’t start run immediately on NOTIFY like they do if the minimum buffer size is at least 1. The problem appears to be the job_available message resolver dropping jobs when the buffer has no room to accept any.

If I am understanding the buffer code correctly, maybe JobBuffer#accept? should always allow up to waiting_count jobs to be accepted? Alternatively, special case when the buffer size is 0 to allow one job if there’s any workers waiting?

[ZimbiX]

Wow, this is certainly worth looking into.

[ZimbiX]

Confirmed on v2.0.0.beta1. Yikes, since we're using a buffer size of 0 in our apps, this means our notify isn't working at all and is just wasting CPU!

With app.rb:

require 'bundler/setup'

require 'sequel'
require 'que'

DB = Sequel.connect(ENV['DATABASE_URL'])
Que.connection = DB

class MyJob < Que::Job
  def run
    puts 'Starting MyJob'
    sleep 2
    puts 'Finished MyJob'
  end
end

and polling disabled:

diff --git a/lib/que/locker.rb b/lib/que/locker.rb
index 013393b..cd6d20a 100644
--- a/lib/que/locker.rb
+++ b/lib/que/locker.rb
@@ -282,7 +282,7 @@ module Que
     def cycle
       # Poll at the start of a cycle, so that when the worker starts up we can
       # load up the queue with jobs immediately.
-      poll
+      # poll
 
       # If we got the stop call while we were polling, break before going to
       # sleep.

and using the database from Docker Compose:

docker-compose up -d db
export DATABASE_URL=postgres://que:que@localhost:5697/que-test

Running Que in one terminal:

ruby -r ./app.rb -e 'DB[:que_jobs].truncate' && \
  bundle exec que --worker-count=5 --maximum-buffer-size=0 --log-level=debug --log-internals ./app.rb \
  | grep -v listener_waiting

Enqueue a job in another terminal:

ruby -r ./app.rb -e 'MyJob.enqueue'

MyJob never runs!

Test results from enqueueing 5 jobs at once:

Worker threads	Buffer size	Jobs run	Working correctly
1	0	0	No
5	0	0	No
1	1	2	Yes
2	1	3	Yes
3	1	4	Yes
1	2	3	Yes
1	3	4	Yes

Edit: Actually this doesn't affect us at GreenSync right now - we moved away from a buffer size of 0 a while ago when we started using multi-threaded workers (after we found a solution to que-locks/que-locks#3)