async producer not to circuit beak on partitioner errors

[ssahukar]

If Partitioner.Partition() function returns an error, the circuit breaker breaks the circuit and drops some of the subsequent messages. I think Sarama should only drop the message for which partitioner returned an error and send that error to the application through the error channel.

In our application with a custom partitioner, we want to drop the messages that get produced with an invalid key, in which case the expected behavior is to drop only that message but to continue processing the other messages as usual, without a circuit break.

[wvanbergen]

The reason the circuit breaker was added is the case of messages being produced to a topic that does not exists. This will fail very slowly, due to the fact that the error (UnknownTopicOrPartition) can also legitimately be temporary, so it needs to be retried.

So the circuit breaker was added, but we didn't really think of other use cases. Maybe we can fix the above problem in a better way.

[eapache]

@ssahukar this should be fixed in the latest master, if I understood you correctly

[ssahukar]

Thanks Evan.

I am following the discussion and it seems it would fix the issue I am
seeing.

Appreciate the quick turn around on this.

On 9 June 2015 at 11:28, Evan Huus notifications@github.com wrote:

@ssahukar https://github.com/ssahukar this should be fixed in the
latest master, if I understood you correctly

—
Reply to this email directly or view it on GitHub
#466 (comment).

[helayzhang]

The reason the circuit breaker was added is the case of messages being produced to a topic that does not exists. This will fail very slowly, due to the fact that the error (UnknownTopicOrPartition) can also legitimately be temporary, so it needs to be retried.

So the circuit breaker was added, but we didn't really think of other use cases. Maybe we can fix the above problem in a better way.

@wvanbergen @eapache

I have met a problem recently. My kafka has disabled automatic topic creation, once a lot of illegal
messages try to write to a not exist topic, circuit breaker is soon opened, then other legal messages will be dropped also.

I'm confusing that in which case UnknownTopicOrPartition may be temporary, so we need circuit breaker to solve this? Hope you to help me understand this better, thx.