Skip to content

[Enhancement](multi-catalog) Add PredicateFilterTime, DictFilterRewriteTime, LazyReadFilteredRows profile metrics in parquet orc profiles. #51248

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
May 28, 2025
Merged

[Enhancement](multi-catalog) Add PredicateFilterTime, DictFilterRewriteTime, LazyReadFilteredRows profile metrics in parquet orc profiles. #51248

morningman merged 1 commit into from
May 28, 2025

Conversation

@kaka11chen
Copy link
Contributor

@kaka11chen kaka11chen commented May 26, 2025 •
edited
Loading

What problem does this PR solve?

Problem Summary:

Release note

[Enhancement] (multi-catalog) Add PredicateFilterTime, DictFilterRewriteTime, LazyReadFilteredRows profile metrics in parquet orc profiles.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented May 26, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@kaka11chen kaka11chen force-pushed the add_parquet_orc_profiles branch from 28d02a0 to c55027b Compare May 26, 2025 09:14
@kaka11chen
Copy link
Contributor Author

kaka11chen commented May 26, 2025

run buildall

@kaka11chen kaka11chen force-pushed the add_parquet_orc_profiles branch from c55027b to 005238f Compare May 26, 2025 09:28
@kaka11chen
Copy link
Contributor Author

kaka11chen commented May 26, 2025

run buildall

@kaka11chen
[Enhancement](multi-catalog) Add PredicateFilterTime, DictFilterRewri…
17dd58b 
…teTime, LazyReadFilteredRows profile metrics in parquet orc profiles.
@kaka11chen kaka11chen force-pushed the add_parquet_orc_profiles branch from 005238f to 17dd58b Compare May 26, 2025 09:48
@kaka11chen
Copy link
Contributor Author

kaka11chen commented May 26, 2025

run buildall

@doris-robot
Copy link

doris-robot commented May 26, 2025

TPC-H: Total hot run time: 33733 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 17dd58baa218eb83365179f0f1c4eb956c28adaf, data reload: false

------ Round 1 ----------------------------------
q1	26581	5072	4995	4995
q2	2075	288	198	198
q3	10374	1242	699	699
q4	10243	1002	520	520
q5	7539	2342	2338	2338
q6	181	164	132	132
q7	918	743	608	608
q8	9322	1270	1066	1066
q9	6869	5156	5092	5092
q10	6881	2320	1918	1918
q11	509	287	271	271
q12	336	360	215	215
q13	17770	3677	3051	3051
q14	239	225	215	215
q15	534	478	494	478
q16	427	437	373	373
q17	611	865	365	365
q18	7787	7289	7108	7108
q19	3190	951	546	546
q20	339	319	224	224
q21	3636	2577	2331	2331
q22	1010	1007	990	990
Total cold run time: 117371 ms
Total hot run time: 33733 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5300	5212	5153	5153
q2	235	323	232	232
q3	2173	2670	2235	2235
q4	1355	1818	1474	1474
q5	4477	4462	4359	4359
q6	217	172	125	125
q7	1991	1990	1816	1816
q8	2659	2587	2475	2475
q9	7204	7093	7089	7089
q10	2992	3191	2771	2771
q11	581	500	501	500
q12	700	785	636	636
q13	3522	3984	3339	3339
q14	283	303	292	292
q15	545	489	477	477
q16	444	479	471	471
q17	1207	1530	1395	1395
q18	7807	7568	7191	7191
q19	833	822	794	794
q20	1987	2064	1900	1900
q21	4901	4484	4339	4339
q22	1106	1051	1037	1037
Total cold run time: 52519 ms
Total hot run time: 50100 ms

@doris-robot
Copy link

doris-robot commented May 26, 2025

TPC-DS: Total hot run time: 191426 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 17dd58baa218eb83365179f0f1c4eb956c28adaf, data reload: false

query1	1407	1136	1059	1059
query2	6292	1798	1812	1798
query3	11127	4650	4710	4650
query4	57702	25131	22906	22906
query5	4791	503	444	444
query6	387	202	226	202
query7	5033	505	289	289
query8	326	242	238	238
query9	6803	2578	2568	2568
query10	421	322	264	264
query11	14970	14971	14758	14758
query12	161	108	106	106
query13	1160	546	395	395
query14	10067	6300	6369	6300
query15	222	196	185	185
query16	7059	675	512	512
query17	1036	729	614	614
query18	1565	422	306	306
query19	202	182	160	160
query20	125	124	124	124
query21	204	124	112	112
query22	4781	4644	4431	4431
query23	33963	33612	33484	33484
query24	6561	2471	2460	2460
query25	474	473	408	408
query26	701	274	147	147
query27	2271	523	347	347
query28	2952	2098	2093	2093
query29	563	564	453	453
query30	274	229	199	199
query31	856	868	799	799
query32	88	70	68	68
query33	472	405	329	329
query34	786	892	543	543
query35	794	850	771	771
query36	989	991	892	892
query37	108	98	74	74
query38	4264	4321	4176	4176
query39	1478	1405	1406	1405
query40	205	116	102	102
query41	55	52	54	52
query42	127	109	110	109
query43	496	499	477	477
query44	1288	818	819	818
query45	186	165	160	160
query46	824	1017	641	641
query47	1817	1820	1794	1794
query48	390	413	332	332
query49	694	501	409	409
query50	668	711	401	401
query51	4153	4147	4091	4091
query52	111	109	103	103
query53	224	248	180	180
query54	571	572	509	509
query55	86	86	78	78
query56	294	290	287	287
query57	1136	1149	1087	1087
query58	265	250	246	246
query59	2617	2717	2570	2570
query60	327	305	306	305
query61	127	123	120	120
query62	741	718	644	644
query63	221	187	195	187
query64	1454	1036	662	662
query65	4313	4232	4219	4219
query66	708	407	306	306
query67	15792	15578	15421	15421
query68	4855	899	527	527
query69	493	304	264	264
query70	1227	1125	1070	1070
query71	432	326	298	298
query72	5781	4858	4877	4858
query73	789	671	355	355
query74	8835	9128	8622	8622
query75	3330	3173	2663	2663
query76	3355	1208	763	763
query77	555	372	284	284
query78	10111	9961	9241	9241
query79	2446	794	573	573
query80	804	513	459	459
query81	516	258	221	221
query82	397	125	97	97
query83	259	248	239	239
query84	289	97	89	89
query85	765	351	321	321
query86	393	311	302	302
query87	4377	4399	4229	4229
query88	3207	2316	2310	2310
query89	442	318	294	294
query90	1969	209	201	201
query91	138	141	113	113
query92	78	58	54	54
query93	2820	914	572	572
query94	675	396	355	355
query95	368	295	290	290
query96	511	584	287	287
query97	2724	2783	2621	2621
query98	234	205	198	198
query99	1302	1416	1277	1277
Total cold run time: 297802 ms
Total hot run time: 191426 ms

@doris-robot
Copy link

doris-robot commented May 26, 2025

ClickBench: Total hot run time: 28.99 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 17dd58baa218eb83365179f0f1c4eb956c28adaf, data reload: false

query1	0.04	0.03	0.04
query2	0.13	0.11	0.11
query3	0.26	0.19	0.19
query4	1.60	0.20	0.10
query5	0.44	0.42	0.43
query6	1.16	0.67	0.65
query7	0.03	0.02	0.02
query8	0.05	0.04	0.03
query9	0.58	0.53	0.52
query10	0.58	0.58	0.56
query11	0.16	0.11	0.11
query12	0.14	0.11	0.11
query13	0.62	0.61	0.60
query14	0.80	0.80	0.82
query15	0.90	0.89	0.86
query16	0.38	0.38	0.39
query17	1.03	1.04	1.04
query18	0.22	0.21	0.21
query19	1.98	1.84	1.79
query20	0.01	0.01	0.02
query21	15.40	0.88	0.54
query22	0.77	1.09	0.66
query23	15.00	1.39	0.64
query24	6.68	1.87	0.77
query25	0.45	0.20	0.09
query26	0.65	0.16	0.14
query27	0.05	0.06	0.05
query28	9.83	0.89	0.46
query29	12.55	3.97	3.27
query30	0.25	0.09	0.06
query31	2.83	0.60	0.39
query32	3.23	0.55	0.47
query33	3.06	3.10	3.06
query34	15.80	5.19	4.55
query35	4.59	4.56	4.51
query36	0.66	0.49	0.47
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.18	0.13	0.13
query41	0.09	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 103.42 s
Total hot run time: 28.99 s

@hello-stephen
Copy link
Contributor

hello-stephen commented May 26, 2025

BE UT Coverage Report

Increment line coverage 9.46% (14/148) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 56.06% (14937/26645)
Line Coverage 44.85% (132944/296450)
Region Coverage 43.91% (66812/152155)
Branch Coverage 38.54% (34279/88954)

@hello-stephen
Copy link
Contributor

hello-stephen commented May 26, 2025

BE Regression && UT Coverage Report

Increment line coverage 54.73% (81/148) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.49% (20848/26228)
Line Coverage 72.71% (215531/296437)
Region Coverage 70.85% (126590/178662)
Branch Coverage 64.64% (65658/101580)

1 similar comment
@hello-stephen
Copy link
Contributor

hello-stephen commented May 26, 2025

BE Regression && UT Coverage Report

Increment line coverage 54.73% (81/148) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.49% (20848/26228)
Line Coverage 72.71% (215531/296437)
Region Coverage 70.85% (126590/178662)
Branch Coverage 64.64% (65658/101580)

@kaka11chen kaka11chen marked this pull request as ready for review May 27, 2025 01:56
@github-actions
Copy link
Contributor

github-actions bot commented May 28, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels May 28, 2025
@github-actions
Copy link
Contributor

github-actions bot commented May 28, 2025

PR approved by anyone and no changes requested.

@morningman morningman merged commit 9745bef into apache:master May 28, 2025
27 of 29 checks passed
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@kaka11chen @koarz
[Enhancement](multi-catalog) Add PredicateFilterTime, DictFilterRewri…
48ea89a 
…teTime, LazyReadFilteredRows profile metrics in parquet orc profiles. (apache#51248)

[Enhancement] (multi-catalog) Add PredicateFilterTime,
DictFilterRewriteTime, LazyReadFilteredRows profile metrics in parquet/orc profiles.
kaka11chen added a commit to kaka11chen/doris that referenced this pull request Jul 1, 2025
@kaka11chen
[Enhancement](multi-catalog) Add PredicateFilterTime, DictFilterRewri…
f184a27 
…teTime, LazyReadFilteredRows profile metrics in parquet orc profiles. (apache#51248)

[Enhancement] (multi-catalog) Add PredicateFilterTime,
DictFilterRewriteTime, LazyReadFilteredRows profile metrics in parquet/orc profiles.
@kaka11chen kaka11chen mentioned this pull request Jul 1, 2025
Merged
morrySnow pushed a commit that referenced this pull request Jul 2, 2025
@kaka11chen
branch-3.1: [Enhancement](multi-catalog) Add PredicateFilterTime, Dic…
efba6cb 
…tFilterRewriteTime, LazyReadFilteredRows profile metrics in parquet orc profiles. #51248 (#52615)

Cherry-pick #51248
kaka11chen added a commit to kaka11chen/doris that referenced this pull request Jul 4, 2025
@kaka11chen
[Enhancement](multi-catalog) Add PredicateFilterTime, DictFilterRewri…
73d2802 
…teTime, LazyReadFilteredRows profile metrics in parquet orc profiles. (apache#51248)

[Enhancement] (multi-catalog) Add PredicateFilterTime,
DictFilterRewriteTime, LazyReadFilteredRows profile metrics in parquet/orc profiles.
@kaka11chen kaka11chen mentioned this pull request Jul 4, 2025
Merged
16 tasks
kaka11chen added a commit to kaka11chen/doris that referenced this pull request Jul 9, 2025
@kaka11chen
[Enhancement](multi-catalog) Add PredicateFilterTime, DictFilterRewri…
c5b0d30 
…teTime, LazyReadFilteredRows profile metrics in parquet orc profiles. (apache#51248)

[Enhancement] (multi-catalog) Add PredicateFilterTime,
DictFilterRewriteTime, LazyReadFilteredRows profile metrics in parquet/orc profiles.
yiguolei pushed a commit that referenced this pull request Jul 16, 2025
@kaka11chen
[Enhancement](multi-catalog) Add PredicateFilterTime, DictFilterRewri…
4b261d4 
…teTime, LazyReadFilteredRows profile metrics in parquet orc profiles. (#52815)

### What problem does this PR solve?

Problem Summary:

### Release note

Cherry-pick #51248

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
kaka11chen added a commit to kaka11chen/doris that referenced this pull request Jul 21, 2025
@kaka11chen
[Enhancement](multi-catalog) Add PredicateFilterTime, DictFilterRewri…
e25ddfd 
…teTime, LazyReadFilteredRows profile metrics in parquet orc profiles.

Cherry-pick apache#51248
@kaka11chen kaka11chen mentioned this pull request Jul 21, 2025
Merged
16 tasks
@morningman morningman added the usercase Important user case type label label Jul 21, 2025
dataroaring pushed a commit that referenced this pull request Jul 22, 2025
@kaka11chen
[Enhancement](multi-catalog) Add PredicateFilterTime, DictFilterRewri…
2b2677c 
…teTime, LazyReadFilteredRows profile metrics in parquet orc profiles. (#53609)

### What problem does this PR solve?

Problem Summary:

### Release note

Cherry-pick #51248
@yiguolei yiguolei mentioned this pull request Aug 11, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

+2 more reviewers

@suxiaogang223 suxiaogang223 suxiaogang223 approved these changes

@hubgeter hubgeter hubgeter approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.11-merged dev/3.0.7-merged dev/3.1.0-merged reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

10 participants

@kaka11chen @Thearas @doris-robot @hello-stephen @morningman @suxiaogang223 @hubgeter @yiguolei @dataroaring @morrySnow