[Bug][Connector-StarRocks] Fix label reuse on stream_load failure causing infinite retry and data loss#10680
Open
xyueji wants to merge 1 commit intoapache:devfrom
Open
[Bug][Connector-StarRocks] Fix label reuse on stream_load failure causing infinite retry and data loss#10680xyueji wants to merge 1 commit intoapache:devfrom
xyueji wants to merge 1 commit intoapache:devfrom
Conversation
…sing infinite retry and data loss Made-with: Cursor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose of this pull request
Fix two issues in StarRocks sink connector when
stream_loadfails with "Label has already been used":Bug 1 - Data loss in
StarRocksSinkManager: When astream_loadfails and StarRocks retains the label, the current code detects "Label has already been used" andbreaks out of the retry loop, silently skipping the entire batch. This causes data loss because the data was never actually loaded into StarRocks.Fix: Replace
breakwith generating a new label andcontinueto retry the batch with the new label.Bug 2 - Missing
reCreateLabelflag inStarRocksStreamLoadVisitor: WhendoStreamLoad()returns Status="Fail" with a message containing "has already been used", the thrownStarRocksConnectorExceptiondoes not setreCreateLabel=true. This prevents the retry logic from generating a new label via theneedReCreateLabel()path.Fix: Detect "has already been used" in the failure message and pass
reCreateLabel=trueto the exception constructor.Does this PR introduce any user-facing change?
No. This is a bug fix for an internal retry mechanism. Users will see that
stream_loadretries now succeed (with a new label) instead of either looping infinitely or silently losing data.How was this patch tested?
Manually tested by reproducing the scenario where StarRocks retains a failed label. After the fix, the retry generates a new label and completes successfully without data loss.
Check list
release-note.