[Optimize](orc-reader) Optimize stripe footer multiple reads in orc reader. #51117

kaka11chen · 2025-05-21T07:49:46Z

What problem does this PR solve?

Related PR: apache/doris-thirdparty#315 apache/doris-thirdparty#316

Problem Summary:

Release note

[Optimize] Optimize stripe footer multiple reads.
Each time a StripeInformationImpl is created, the stripe footer is loaded and read lazily.
Fix this problem by adding the ability to pass in the stripe footer that has been read in the StripeInformationImpl constructor.

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

kaka11chen · 2025-05-21T07:49:58Z

run buildall

Thearas · 2025-05-21T07:50:05Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

doris-robot · 2025-05-21T09:40:59Z

TPC-H: Total hot run time: 33814 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e48e1206e8682f19f6516e05f1e126fc2d27b7db, data reload: false

------ Round 1 ----------------------------------
q1	26460	5066	5020	5020
q2	2080	297	192	192
q3	10592	1294	693	693
q4	10231	998	516	516
q5	7755	2340	2365	2340
q6	180	160	131	131
q7	930	744	607	607
q8	9318	1224	1089	1089
q9	6876	5064	5083	5064
q10	6861	2313	1887	1887
q11	493	296	274	274
q12	349	341	216	216
q13	17777	3645	3084	3084
q14	231	231	218	218
q15	535	497	495	495
q16	419	433	378	378
q17	602	861	360	360
q18	7695	7211	7128	7128
q19	1362	926	567	567
q20	344	332	234	234
q21	3882	3158	2330	2330
q22	1062	1052	991	991
Total cold run time: 116034 ms
Total hot run time: 33814 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5064	5048	5037	5037
q2	241	327	227	227
q3	2173	2639	2278	2278
q4	1415	1748	1404	1404
q5	4461	4365	4418	4365
q6	212	176	130	130
q7	2030	1912	1783	1783
q8	2564	2584	2515	2515
q9	7244	7256	6969	6969
q10	3100	3210	2762	2762
q11	579	517	503	503
q12	678	769	621	621
q13	3523	3871	3261	3261
q14	278	307	266	266
q15	528	491	486	486
q16	457	482	440	440
q17	1129	1530	1363	1363
q18	7942	7610	7499	7499
q19	805	808	871	808
q20	2019	2044	1860	1860
q21	4865	4535	4423	4423
q22	1127	1072	1042	1042
Total cold run time: 52434 ms
Total hot run time: 50042 ms

doris-robot · 2025-05-21T09:53:14Z

TPC-DS: Total hot run time: 193350 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e48e1206e8682f19f6516e05f1e126fc2d27b7db, data reload: false

query1	1430	1096	1063	1063
query2	6340	1869	1851	1851
query3	11030	4474	4599	4474
query4	54450	25189	23579	23579
query5	5057	572	461	461
query6	330	216	221	216
query7	4896	511	296	296
query8	311	260	237	237
query9	5483	2639	2657	2639
query10	463	321	287	287
query11	15031	14951	14792	14792
query12	167	113	106	106
query13	1059	525	429	429
query14	10282	6333	6366	6333
query15	205	212	179	179
query16	7122	665	471	471
query17	1089	754	620	620
query18	1652	428	320	320
query19	207	191	176	176
query20	141	128	129	128
query21	207	129	112	112
query22	4278	4421	4314	4314
query23	34627	33654	33496	33496
query24	6689	2438	2460	2438
query25	482	488	440	440
query26	705	268	153	153
query27	2275	508	348	348
query28	2993	2189	2205	2189
query29	587	565	439	439
query30	290	214	190	190
query31	847	867	788	788
query32	82	65	66	65
query33	450	378	338	338
query34	782	873	540	540
query35	801	842	747	747
query36	966	1022	897	897
query37	109	98	80	80
query38	4304	4347	4205	4205
query39	1531	1473	1487	1473
query40	223	125	107	107
query41	64	56	54	54
query42	129	114	109	109
query43	524	526	456	456
query44	1374	839	835	835
query45	183	176	163	163
query46	859	1053	672	672
query47	1810	1882	1780	1780
query48	421	440	330	330
query49	664	537	425	425
query50	669	694	408	408
query51	4298	4334	4225	4225
query52	114	107	107	107
query53	225	260	187	187
query54	601	580	522	522
query55	88	89	88	88
query56	314	338	296	296
query57	1170	1191	1134	1134
query58	282	281	259	259
query59	2696	2832	2803	2803
query60	354	324	301	301
query61	129	128	135	128
query62	744	750	695	695
query63	215	182	195	182
query64	1804	1058	691	691
query65	4289	4269	4196	4196
query66	726	404	302	302
query67	16195	15889	15387	15387
query68	6898	889	520	520
query69	530	316	271	271
query70	1222	1178	1095	1095
query71	517	320	314	314
query72	5822	4702	4843	4702
query73	1469	626	354	354
query74	9046	9114	9007	9007
query75	3958	3218	2702	2702
query76	4302	1190	747	747
query77	723	354	294	294
query78	10182	10282	9361	9361
query79	2386	823	588	588
query80	613	512	443	443
query81	479	266	220	220
query82	433	129	103	103
query83	304	248	236	236
query84	292	107	87	87
query85	788	348	309	309
query86	357	304	305	304
query87	4406	4449	4319	4319
query88	3463	2330	2312	2312
query89	478	316	282	282
query90	1910	201	206	201
query91	145	145	112	112
query92	78	56	57	56
query93	1698	921	576	576
query94	664	418	367	367
query95	368	289	296	289
query96	501	568	291	291
query97	2778	2751	2662	2662
query98	232	220	213	213
query99	1458	1385	1262	1262
Total cold run time: 299414 ms
Total hot run time: 193350 ms

doris-robot · 2025-05-21T09:58:25Z

ClickBench: Total hot run time: 29.63 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e48e1206e8682f19f6516e05f1e126fc2d27b7db, data reload: false

query1	0.03	0.03	0.03
query2	0.12	0.10	0.12
query3	0.27	0.19	0.19
query4	1.59	0.19	0.19
query5	0.45	0.44	0.45
query6	1.58	0.67	0.67
query7	0.02	0.01	0.01
query8	0.05	0.04	0.03
query9	0.58	0.53	0.52
query10	0.58	0.58	0.57
query11	0.16	0.11	0.10
query12	0.14	0.11	0.12
query13	0.61	0.60	0.60
query14	0.79	0.80	0.82
query15	0.87	0.86	0.85
query16	0.39	0.39	0.39
query17	1.05	1.07	1.02
query18	0.22	0.22	0.21
query19	1.95	1.78	1.81
query20	0.01	0.01	0.02
query21	15.44	0.88	0.56
query22	0.76	1.28	0.73
query23	14.71	1.37	0.61
query24	6.96	1.42	1.18
query25	0.50	0.18	0.12
query26	0.58	0.16	0.15
query27	0.05	0.06	0.06
query28	9.76	0.88	0.45
query29	12.55	4.02	3.37
query30	0.25	0.10	0.06
query31	2.82	0.58	0.39
query32	3.23	0.55	0.46
query33	3.02	3.12	3.04
query34	15.74	5.11	4.54
query35	4.52	4.53	4.49
query36	0.65	0.50	0.48
query37	0.09	0.07	0.06
query38	0.05	0.03	0.04
query39	0.02	0.02	0.02
query40	0.18	0.14	0.14
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 103.48 s
Total hot run time: 29.63 s

…eader.

kaka11chen · 2025-05-21T10:57:10Z

run buildall

doris-robot · 2025-05-21T12:47:58Z

TPC-H: Total hot run time: 34269 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5d33762cf285fb0520e7db0872d0e8fae375d2f0, data reload: false

------ Round 1 ----------------------------------
q1	26464	5089	5298	5089
q2	2096	285	188	188
q3	10587	1295	706	706
q4	10240	1042	530	530
q5	7876	2479	2345	2345
q6	189	169	135	135
q7	937	760	628	628
q8	9322	1338	1113	1113
q9	6747	5156	5106	5106
q10	6891	2315	1915	1915
q11	480	309	279	279
q12	366	381	216	216
q13	17839	3737	3209	3209
q14	236	230	220	220
q15	542	486	490	486
q16	425	437	379	379
q17	639	851	408	408
q18	7596	7259	7154	7154
q19	1456	961	577	577
q20	344	344	228	228
q21	4128	3490	2430	2430
q22	1031	1003	928	928
Total cold run time: 116431 ms
Total hot run time: 34269 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5251	5179	5053	5053
q2	238	324	227	227
q3	2171	2683	2337	2337
q4	1388	1889	1415	1415
q5	4518	4458	4390	4390
q6	221	177	130	130
q7	2085	1921	1782	1782
q8	2636	2548	2565	2548
q9	7203	7165	7033	7033
q10	3023	3192	2783	2783
q11	580	515	486	486
q12	708	757	622	622
q13	3539	3962	3346	3346
q14	290	293	291	291
q15	546	500	472	472
q16	433	486	435	435
q17	1171	1568	1412	1412
q18	7764	7519	7492	7492
q19	849	846	905	846
q20	2000	2119	1868	1868
q21	4878	4570	4477	4477
q22	1066	1075	1041	1041
Total cold run time: 52558 ms
Total hot run time: 50486 ms

doris-robot · 2025-05-21T13:00:16Z

TPC-DS: Total hot run time: 193000 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5d33762cf285fb0520e7db0872d0e8fae375d2f0, data reload: false

query1	1432	1147	1076	1076
query2	6160	1828	1808	1808
query3	11002	4654	4589	4589
query4	54023	26185	22914	22914
query5	5308	523	495	495
query6	373	230	215	215
query7	5033	536	304	304
query8	332	283	252	252
query9	6495	2653	2650	2650
query10	448	352	297	297
query11	15299	15178	14783	14783
query12	161	113	109	109
query13	1141	532	419	419
query14	10337	6378	6425	6378
query15	206	203	180	180
query16	7157	683	513	513
query17	1102	763	616	616
query18	1569	425	351	351
query19	203	192	172	172
query20	126	124	122	122
query21	207	126	106	106
query22	4249	4348	4296	4296
query23	34276	33658	33446	33446
query24	6768	2472	2510	2472
query25	478	479	429	429
query26	695	285	163	163
query27	2247	520	357	357
query28	2964	2168	2155	2155
query29	594	578	446	446
query30	269	222	196	196
query31	898	847	811	811
query32	76	67	61	61
query33	463	384	340	340
query34	815	898	583	583
query35	799	837	744	744
query36	1037	1073	905	905
query37	119	101	81	81
query38	4266	4293	4245	4245
query39	1515	1452	1672	1452
query40	210	134	111	111
query41	57	54	55	54
query42	131	109	125	109
query43	509	531	471	471
query44	1446	858	849	849
query45	189	180	172	172
query46	886	1059	663	663
query47	1823	1875	1780	1780
query48	417	476	336	336
query49	702	551	464	464
query50	702	725	425	425
query51	4225	4348	4218	4218
query52	129	110	109	109
query53	247	273	191	191
query54	607	593	536	536
query55	88	87	86	86
query56	310	319	302	302
query57	1200	1209	1118	1118
query58	285	265	263	263
query59	2718	2705	2669	2669
query60	357	334	327	327
query61	154	127	123	123
query62	753	744	703	703
query63	243	206	196	196
query64	1700	1064	678	678
query65	4397	4239	4243	4239
query66	739	406	309	309
query67	15903	15690	15436	15436
query68	7114	886	528	528
query69	544	317	277	277
query70	1189	1148	1117	1117
query71	512	339	308	308
query72	5713	4672	4645	4645
query73	1312	586	354	354
query74	9210	9163	9053	9053
query75	3767	3236	2726	2726
query76	4221	1208	758	758
query77	614	388	304	304
query78	10098	10235	9300	9300
query79	2610	782	589	589
query80	684	502	550	502
query81	485	253	220	220
query82	436	125	97	97
query83	369	261	237	237
query84	293	109	84	84
query85	778	352	311	311
query86	403	312	271	271
query87	4371	4410	4357	4357
query88	3424	2341	2302	2302
query89	409	316	281	281
query90	1787	208	205	205
query91	150	153	116	116
query92	73	63	52	52
query93	2236	962	575	575
query94	657	411	308	308
query95	373	295	288	288
query96	496	592	291	291
query97	2744	2800	2618	2618
query98	228	211	203	203
query99	1451	1446	1255	1255
Total cold run time: 300520 ms
Total hot run time: 193000 ms

doris-robot · 2025-05-21T13:05:29Z

ClickBench: Total hot run time: 29.28 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5d33762cf285fb0520e7db0872d0e8fae375d2f0, data reload: false

query1	0.04	0.03	0.03
query2	0.13	0.11	0.11
query3	0.26	0.19	0.20
query4	1.60	0.20	0.20
query5	0.46	0.44	0.44
query6	1.14	0.67	0.65
query7	0.03	0.02	0.02
query8	0.04	0.04	0.04
query9	0.60	0.53	0.53
query10	0.58	0.58	0.58
query11	0.16	0.11	0.12
query12	0.14	0.11	0.11
query13	0.62	0.60	0.60
query14	0.79	0.79	0.82
query15	0.88	0.85	0.86
query16	0.38	0.37	0.38
query17	1.08	1.09	1.05
query18	0.23	0.21	0.21
query19	1.94	1.88	1.85
query20	0.01	0.01	0.02
query21	15.42	0.93	0.56
query22	0.76	1.12	0.61
query23	15.02	1.40	0.61
query24	7.23	1.67	0.85
query25	0.49	0.27	0.08
query26	0.54	0.16	0.14
query27	0.05	0.05	0.06
query28	9.58	0.86	0.45
query29	12.58	3.94	3.42
query30	0.24	0.09	0.06
query31	2.84	0.57	0.39
query32	3.27	0.55	0.46
query33	3.05	3.09	3.04
query34	15.85	5.07	4.51
query35	4.50	4.51	4.54
query36	0.66	0.51	0.48
query37	0.09	0.07	0.06
query38	0.06	0.04	0.04
query39	0.02	0.02	0.03
query40	0.16	0.14	0.13
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 103.67 s
Total hot run time: 29.28 s

hello-stephen · 2025-05-21T14:49:20Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	55.96% (14927/26676)
Line Coverage	44.77% (132401/295758)
Region Coverage	43.86% (66618/151889)
Branch Coverage	38.44% (34133/88784)

hello-stephen · 2025-05-21T16:44:30Z

BE Regression && UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.35% (20837/26261)
Line Coverage	72.56% (214593/295756)
Region Coverage	70.77% (126294/178445)
Branch Coverage	64.50% (65411/101420)

hello-stephen · 2025-05-22T02:04:53Z

BE Regression && UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.35% (20837/26261)
Line Coverage	72.56% (214593/295756)
Region Coverage	70.77% (126294/178445)
Branch Coverage	64.50% (65411/101420)

github-actions · 2025-05-24T02:57:48Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-05-24T02:57:50Z

PR approved by anyone and no changes requested.

…eader. (apache#51117) ### What problem does this PR solve? Related PR: apache/doris-thirdparty#315 apache/doris-thirdparty#316

…eader. (#51277) ### What problem does this PR solve? Problem Summary: Cherry-pick #51117 None ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

[Optimize](orc-reader) Optimize stripe footer multiple reads in orc r…

5d33762

…eader.

kaka11chen force-pushed the opt_stripe_footer_multi_reading branch from e48e120 to 5d33762 Compare May 21, 2025 10:57

morningman added dev/2.1.x dev/3.0.x usercase Important user case type label labels May 23, 2025

morningman approved these changes May 24, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label May 24, 2025

github-actions bot added the reviewed label May 24, 2025

CalvinKirs approved these changes May 24, 2025

View reviewed changes

morningman merged commit 22d2932 into apache:master May 24, 2025
31 of 33 checks passed

github-actions bot added dev/3.0.x-conflict dev/2.1.x-conflict labels May 24, 2025

This was referenced May 27, 2025

[Optimize](orc-reader) Optimize stripe footer multiple reads in orc reader. #51276

Merged

[Optimize](orc-reader) Optimize stripe footer multiple reads in orc reader. #51277

Merged

yiguolei removed dev/2.1.x dev/2.1.x-conflict labels May 28, 2025

yiguolei added the dev/2.1.11-merged label May 28, 2025

koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025

[Optimize](orc-reader) Optimize stripe footer multiple reads in orc r…

24398b2

…eader. (apache#51117) ### What problem does this PR solve? Related PR: apache/doris-thirdparty#315 apache/doris-thirdparty#316

dataroaring added dev/3.0.7-merged and removed dev/3.0.x dev/3.0.x-conflict labels Jun 11, 2025

morrySnow added the dev/3.1.0-merged label Jun 17, 2025

yiguolei mentioned this pull request Aug 11, 2025

2.1.11 Release Notes #54550

Open

[Optimize](orc-reader) Optimize stripe footer multiple reads in orc reader. #51117

[Optimize](orc-reader) Optimize stripe footer multiple reads in orc reader. #51117

Uh oh!

Conversation

kaka11chen commented May 21, 2025 • edited by morningman Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

kaka11chen commented May 21, 2025

Uh oh!

Thearas commented May 21, 2025

Uh oh!

doris-robot commented May 21, 2025

Uh oh!

doris-robot commented May 21, 2025

Uh oh!

doris-robot commented May 21, 2025

Uh oh!

kaka11chen commented May 21, 2025

Uh oh!

doris-robot commented May 21, 2025

Uh oh!

doris-robot commented May 21, 2025

Uh oh!

doris-robot commented May 21, 2025

Uh oh!

hello-stephen commented May 21, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented May 21, 2025

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented May 22, 2025

BE Regression && UT Coverage Report

Uh oh!

github-actions bot commented May 24, 2025

Uh oh!

github-actions bot commented May 24, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

kaka11chen commented May 21, 2025 •

edited by morningman

Loading