[v25.2.x] kafka: add kafka_produce_batch_validation and rework timestamp validation (MANUAL BACKPORT)#27599
Merged
WillemKauf merged 5 commits intoredpanda-data:v25.2.xfrom Sep 17, 2025
Conversation
``` Cluster config name: `kafka_produce_batch_validation` Description: Controls the amount of validation that happens for produced batches in the Kafka API. See the below descriptions for the behavior of each of these modes. Type enum: * `legacy`: In legacy mode we do the exact same behavior as we do today in Redpanda 25.2.x - there is minimal validation on the produce path of crc + uncompressed record iteration. * `relaxed`: The new default mode that does full validation of uncompressed batches, this includes setting the max_timestamp. For compressed batches, it does only crc validation EXCEPT if the max_timestamp is missing. In the missing max_timestamp case we decompress the batch to compute the max_timestamp and do full validation. * `strict`: Always do full validation of compressed and uncompressed batches. This should be the default in environments where producing clients are not trusted. Default: `relaxed` ``` (cherry picked from commit 146f1b1)
rockwotj
approved these changes
Sep 16, 2025
WillemKauf
commented
Sep 16, 2025
| {.batch = *batch, | ||
| .timestamp_type = cfg_ctx.timestamp_type, | ||
| .message_timestamp_alert_before_ms | ||
| = config::shard_local_cfg().log_message_timestamp_alert_before_ms, |
Contributor
Author
There was a problem hiding this comment.
note: this PR is using alerts rather than outright rejection of batches with message.timestamp.{before/after}.max.ms
WillemKauf
force-pushed
the
kafka_batch_validation_backport
branch
3 times, most recently
from
6fb2bed to
26ecc26
Compare
(cherry picked from commit 983fb8e)
This commit reworks and adds several new batch/record level validation functions
to the Kafka produce path in `produce_validation.{cc/h}`.
`redpanda` cannot leave `max_timestamp` unset in record batches - several
subsystems (timequeries, retention, archival) depend heavily on the `max_timestamp`
of batches being properly set.
Set the `max_timestamp` manually by iterating over the records in the
record batch, taking the maximum `timestamp_delta` and adding it to the
batch's `first_timestamp` (which _must_ be set by clients, as a hard rule).
This may incur additional cost for compressed batches, and is gated by the
level of `kafka_produce_batch_validation`.
Log lines are added to alert the client in case of a batch accepted without a
maximum timestamp set for `kafka_produce_batch_validation == legacy` or in the case of
expensive decompression operation in the produce path for
`kafka_produce_batch_validation == relaxed`, letting them know to update
their client to set the `max_timestamp`.
(cherry picked from commit 7f148e9)
Tests that records/batches produced with specific timestamps, `redpanda` validation modes and `compression.type`s have the same behavior with a Kafka broker. (cherry picked from commit 58731ae)
This test case uses an old version of `sarama` which does NOT properly set the max_timestamp in a batch. This version is useful for testing redpanda behavior with improper timestamps and different validation modes. (cherry picked from commit c87baf0)
WillemKauf
force-pushed
the
kafka_batch_validation_backport
branch
from
26ecc26 to
35eb4b0
Compare
Contributor
Author
|
Please let this be the last force push. Lots of missing stuff from |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Manual backport of #27529.
cherry-pickconflicts due to lack of #27419 inv25.2.x.Because of the lack of the above PR and
message.timestamp.{before/after}.max.ms, we still rely on alerts (log_message_timestamp_alert_{before/after}_ms)rather than outright rejecting messages.Fixes #27573
Backports Required
Release Notes
Features
kafka_produce_batch_validationwhich controls the level of validation performed on batches in theredpandaproduce path.