Currently we aren't consistently windowing the production of metrics to StatsD from Apache Beam. This means that during batch ingestion, an extremely high volume of metrics are sent to StatsD. This can cause StatsD to become unresponsive or crash.
To make matters worse, these metrics are less useful for batch ingestion than they are for streaming ingestion, so its unexpected that batch users would need these metrics, especially at a low granularity.
The proposed solution is to window the metrics and only push them to StatsD at fixed windows.
Currently we aren't consistently windowing the production of metrics to StatsD from Apache Beam. This means that during batch ingestion, an extremely high volume of metrics are sent to StatsD. This can cause StatsD to become unresponsive or crash.
To make matters worse, these metrics are less useful for batch ingestion than they are for streaming ingestion, so its unexpected that batch users would need these metrics, especially at a low granularity.
The proposed solution is to window the metrics and only push them to StatsD at fixed windows.