Add cluster property to limit max.message.bytes#27997
Merged
ballard26 merged 4 commits intoredpanda-data:devfrom Oct 20, 2025
Merged
Add cluster property to limit max.message.bytes#27997ballard26 merged 4 commits intoredpanda-data:devfrom
max.message.bytes#27997ballard26 merged 4 commits intoredpanda-data:devfrom
Conversation
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR introduces a new cluster property kafka_max_message_size_upper_limit to constrain the maximum value that users can set for the max.message.bytes topic property. The change provides administrators with control over message size limits across all topics.
Key changes:
- Added a new cluster configuration property
kafka_max_message_size_upper_limit - Updated validation logic for
max.message.bytesin both topic creation and configuration alteration - Added comprehensive test coverage for the new validation behavior
Reviewed Changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
src/v/config/configuration.h |
Declares the new kafka_max_message_size_upper_limit property |
src/v/config/configuration.cc |
Defines the configuration property with appropriate metadata |
src/v/kafka/server/handlers/topics/validators.h |
Updates topic creation validator to respect the upper limit |
src/v/kafka/server/handlers/configs/config_utils.h |
Adds new validator for configuration alterations |
src/v/kafka/server/handlers/alter_configs.cc |
Integrates the validator into alter configs flow |
src/v/kafka/server/handlers/incremental_alter_configs.cc |
Integrates the validator into incremental alter configs flow |
tests/rptest/clients/types.py |
Adds constant for the max message bytes property |
tests/rptest/tests/topic_creation_test.py |
Tests validation during topic creation |
tests/rptest/tests/alter_topic_configuration_test.py |
Tests validation during topic configuration changes |
| const uint32_t upper_limit = config::shard_local_cfg() | ||
| .kafka_max_message_size_upper_limit() | ||
| .value_or( | ||
| std::numeric_limits<int32_t>::max()); |
There was a problem hiding this comment.
Type mismatch: upper_limit is declared as uint32_t but std::numeric_limits<int32_t>::max() returns int32_t. This could cause issues when comparing with negative values or when the limit exceeds int32_t range. Either change upper_limit to int32_t or use std::numeric_limits<uint32_t>::max().
Suggested change
| std::numeric_limits<int32_t>::max()); | |
| std::numeric_limits<uint32_t>::max()); |
Comment on lines
+198
to
+201
| auto val = boost::lexical_cast<int32_t>(it->value.value()); | ||
| auto upper_limit = config::shard_local_cfg() | ||
| .kafka_max_message_size_upper_limit() | ||
| .value_or(std::numeric_limits<int32_t>::max()); |
There was a problem hiding this comment.
Type mismatch: upper_limit is inferred as int32_t from value_or() but is compared with val which is int32_t. However, the configuration property is defined as std::optional<int32_t> but used in a context where it might be expected to handle larger values. Consider using consistent types throughout.
Suggested change
| auto val = boost::lexical_cast<int32_t>(it->value.value()); | |
| auto upper_limit = config::shard_local_cfg() | |
| .kafka_max_message_size_upper_limit() | |
| .value_or(std::numeric_limits<int32_t>::max()); | |
| auto val = boost::lexical_cast<int64_t>(it->value.value()); | |
| auto upper_limit = config::shard_local_cfg() | |
| .kafka_max_message_size_upper_limit() | |
| .value_or(std::numeric_limits<int64_t>::max()); |
Contributor
Author
There was a problem hiding this comment.
Yeah, I'm not sure why we're using int32_t for max.message.bytes in the topic creation handler. Then using uint32_t for it in the alter config handler. Maybe an odd nuance with the Kafka API? cc: @michael-redpanda
Collaborator
Retry command for Build#73922please wait until all jobs are finished before running the slash command |
Collaborator
michael-redpanda
previously approved these changes
Oct 10, 2025
Contributor
Author
|
/ci-repeat 1 |
paulohtb6
reviewed
Oct 11, 2025
src/v/config/configuration.cc
Outdated
Comment on lines
+1802
to
+1803
| "Maximum size the server accepts for the `max.message.size` topic " | ||
| "property. If `null`, then no limit is enforced.", |
Contributor
There was a problem hiding this comment.
Suggested change
| "Maximum size the server accepts for the `max.message.size` topic " | |
| "property. If `null`, then no limit is enforced.", | |
| "Maximum allowed value for the `max.message.size` topic " | |
| "property. When set to `null`, no limit is enforced.", |
paulohtb6
reviewed
Oct 11, 2025
src/v/config/configuration.cc
Outdated
| std::nullopt) | ||
| , kafka_max_message_size_upper_limit( | ||
| *this, | ||
| "kafka_max_message_size_upper_limit", |
Contributor
There was a problem hiding this comment.
suggest to rename the property with a suffix on the unit that is being used (bytes, kilobytes, megabytes etc)
ballard26
force-pushed
the
max-message-bytes-limits
branch
from
bf26502 to
de94a64
Compare
ballard26
commented
Oct 16, 2025
| "Maximum allowed value for the `max.message.size` topic " | ||
| "property. When set to `null`, then no limit is enforced.", | ||
| {.needs_restart = needs_restart::no, .visibility = visibility::tunable}, | ||
| 100_MiB, |
Contributor
Author
There was a problem hiding this comment.
Changed this from no default to 100MiB as it matches the max size kafka request Redpanda will process by default. And therefore around what the max batch size could be.
Collaborator
Retry command for Build#74304please wait until all jobs are finished before running the slash command |
ballard26
force-pushed
the
max-message-bytes-limits
branch
from
de94a64 to
a280111
Compare
Contributor
Author
|
The test failures in |
ballard26
force-pushed
the
max-message-bytes-limits
branch
from
a280111 to
23b645a
Compare
michael-redpanda
approved these changes
Oct 20, 2025
Contributor
There was a problem hiding this comment.
nit: commit message should say cluster config
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds the
kafka_max_message_size_upper_limitcluster property. This property can be used to limit the max value of themax.message.bytestopic property.Backports Required
Release Notes
Features
kafka_max_message_size_upper_limitcluster property. This property can be used to limit the max value of themax.message.bytestopic property.