[Enhancement] use stream load async return to optimize the performance under high concurrency #53144
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 3 commits
July 12, 2025 12:04
…e better http performance * fix some bug for partition * fix for thrift * fix the thrift exit bug * Revert "[feat](http): use async reply to provide better http performance * ensure free order * [Fix](stream-load) Fix stream load stuck under high concurrency (apache#36772) When the concurrency of streamload exceeds the number of threads in the remote scanner, streamload may get stuck. The reason is that the libevent thread blocks and waits for streamload to complete, and when there is no intersection between the tasks handled by the scanner thread and the libevent thread, it gets stuck. The solution is to convert the synchronous waiting tasks of libevent into asynchronous execution by using callbacks in the streamload executor thread. See merge request: !740" Revert commit d9e74efa762c8161a5ca3df4290bbd0ab896f1ef See merge request: !745" Revert commit 396cb2ec7e0b1a21bc0d7424c627f0d9321884bc
* fix missing ; * fix stream load block See merge request: !855
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
gavinchou
reviewed
Jul 14, 2025
| } | ||
|
|
||
| int StreamLoadAction::on_header(HttpRequest* req) { | ||
| req->mark_send_reply(); |
Contributor
Author
There was a problem hiding this comment.
只有 StreamLoad 请求，另外只是提交 reply 是 async 的，客户端感知仍然是同步的。
gavinchou
reviewed
Jul 14, 2025
| } | ||
|
|
||
| private: | ||
| SendReplyType _send_reply_type = REPLY_SYNC; |
Contributor
There was a problem hiding this comment.
这个初始值是不是没啥用 因为 on_header 把它改写了
Contributor
Author
There was a problem hiding this comment.
还有其他 HttpAction 会用到，他们需要默认是 SYNC 的。
gavinchou
previously approved these changes
Jul 14, 2025
github-actions
bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
Contributor
|
run buildall |
github-actions
bot
removed
the
approved
gavinchou
previously approved these changes
Jul 14, 2025
Contributor
Author
|
run buildall |
github-actions
bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
added 4 commits
July 15, 2025 10:55
…e better http performance * fix some bug for partition * fix for thrift * fix the thrift exit bug * Revert "[feat](http): use async reply to provide better http performance * ensure free order * [Fix](stream-load) Fix stream load stuck under high concurrency (apache#36772) When the concurrency of streamload exceeds the number of threads in the remote scanner, streamload may get stuck. The reason is that the libevent thread blocks and waits for streamload to complete, and when there is no intersection between the tasks handled by the scanner thread and the libevent thread, it gets stuck. The solution is to convert the synchronous waiting tasks of libevent into asynchronous execution by using callbacks in the streamload executor thread. See merge request: !740" Revert commit d9e74efa762c8161a5ca3df4290bbd0ab896f1ef See merge request: !745" Revert commit 396cb2ec7e0b1a21bc0d7424c627f0d9321884bc
* fix missing ; * fix stream load block See merge request: !855
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
Contributor
Author
|
run buildall |
1 similar comment
Contributor
Author
|
run buildall |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 32488 ms |
TPC-DS: Total hot run time: 173992 ms |
ClickBench: Total hot run time: 27.43 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 31950 ms |
TPC-DS: Total hot run time: 173549 ms |
ClickBench: Total hot run time: 26.99 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 32001 ms |
TPC-DS: Total hot run time: 173277 ms |
ClickBench: Total hot run time: 27.01 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #53139
Problem Summary:
Release note
use stream load async return to optimize the performance under high concurrency, we treat
send_replyas callback of exec_plan_framgnet .is_prepare_successto ensure no dead lock aton_header.ctx-> _finish_send_replyensure send_reply only be invoke one time and request is valid. becuase we can't decide if callback will send_reply again when load fail or load timeout.ctx-> _can_send_replyensure send_reply only whenon_headerfail orhandlefinished, or sender will receive broken-pipe when meet data-quality errorCheck List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)