Proposal: Cap simple_aggregation to 500 records per aggregated record in Kinesis Streams output

[m15o]

I’m considering proposing a change to the Kinesis Streams output in Fluent Bit and wanted to get feedback before opening a PR.

When simple_aggregation is enabled, I’m thinking about limiting the number of events per aggregated record to a fixed maximum of 500.

Rationale:
According to AWS Firehose documentation, the maximum number of records that can be deaggregated from a single record is 500:
https://docs.aws.amazon.com/firehose/latest/dev/dynamic-partitioning-multirecord-deaggergation.html

So aligning with this limit would make the behavior safer and more predictable, especially for users who forward data to Firehose.

Open questions:

• Are there known use cases where simple_aggregation is used outside of Firehose, where a different limit would be required?
• If such use cases exist, would it be better to make this limit configurable instead of fixed?
• If not, is a hard cap of 500 acceptable?

If there’s general agreement, I can proceed with a PR (either fixed or configurable depending on feedback).

[patrick-stephens]

Might be worth publicising in the CNCF slack if you haven't already?
https://cloud-native.slack.com/archives/C0APYFHS5LH