Kafka partitions

[amithalle]

Environment

self-hosted (https://develop.sentry.dev/self-hosted/)

What are you trying to accomplish?

Hi!

I'm using sentry 25.10.0 deployed on kubernetes. Deployed using the helm chart https://artifacthub.io/packages/helm/sentry/sentry

using kafka deployed via strimzi operator, and the kafka topics were created by the migration with --create-kafka-topics. all the topics were created with a single partition, I don't know whether by sentry configuration or kafka configuration.

our use case requires us to set sample rate for replays to 100%.
this causes the kafka queue for replays get overflowed, it currently getting well over 700GB with only 2 days retention, making management of brokers and their disks become an issue. In addition, the ingestion of that topic is bottlenecked in a single consumer.

my questions:
will it be possible to change the topics to have multiple partitions?
do the consumers support that?
if the answer for the previous is positive - is it possible to scale the "ingest replay recordings" consumer with multiple replicas (even better - use a horizontal pod autoscaler, even based on CPU)?

thanks a lot!

How are you getting stuck?

We are facing perfomance bottleneck and delays

Where in the product are you?

Replays

Link

No response

DSN

No response

Version

25.10.0

[linear-code]

ENG-7179

[getsantry]

Assigning to @getsentry/support for routing ⏲️

[aldy505]

will it be possible to change the topics to have multiple partitions?
do the consumers support that?
if the answer for the previous is positive - is it possible to scale the "ingest replay recordings" consumer with multiple replicas (even better - use a horizontal pod autoscaler, even based on CPU)?

yes to everything