Allow que to be started without listen/notify

[dtcristo]

Ability to disable listen/notify on CLI resulting in jobs being picked up purely from polling. We have seen issues with listen/notify usage at scale.

[oeoeaio]

Thanks for this! No idea why this is not already configurable. If you can add some specs I'll be happy to approve and merge this. :)

[dtcristo]

@oeoeaio I've updated the commit with a spec for the new option. Ready for review.

[mnbbrown]

Do you think we could get this merged?
@dtcristo / @ZimbiX shout if I can help in anyway.

[ZimbiX]

Sorry for the delay!

Out of curiosity, what issues were you seeing with listen/notify at scale?

[dtcristo]

@oeoeaio Thanks for merging!

Out of curiosity, what issues were you seeing with listen/notify at scale?

Hey @ZimbiX, we have quite a high throughput of jobs in our system. Average of 150 jobs/s peaking to about 700 jobs/s at times. We had issues where listen/notify queue would fill quickly and when full subsequent listen/notify calls would fail. I believe there were issues where long running transactions prevented the queue from being emptied if the transactions covered items in the queue.

I feel listen/notify is ideal for systems with lower job throughput with sporadic events and you need low latency. In a system with high throughput like ours, poll mode works fine.

[mnbbrown]

@oeoeaio Thanks for merging!

Out of curiosity, what issues were you seeing with listen/notify at scale?

Hey @ZimbiX, we have quite a high throughput of jobs in our system. Average of 150 jobs/s peaking to about 700 jobs/s at times. We had issues where listen/notify queue would fill quickly and when full subsequent listen/notify calls would fail. I believe there were issues where long running transactions prevented the queue from being emptied if the transactions covered items in the queue.

I feel listen/notify is ideal for systems with lower job throughput with sporadic events and you need low latency. In a system with high throughput like ours, poll mode works fine.

For us, we're experimenting with pgcat and it'd doesn't yet support LISTEN/NOTIFY so this is a good fallback :)