[fix](statistics)Fix sample analyze agg/mor table doesn't remove new partition flag bug. #51116

Jibing-Li · 2025-05-21T07:47:28Z

What problem does this PR solve?

Related PR: #49918

Problem Summary:

Fix sample analyze agg/mor table doesn't remove new partition flag bug.
When a new partition load data for the first time, we set a flag to this table's metadata, this flag indicate that the statistics data of this table is expired and not accurate, so it will trigger a new collection for this table. After collect all columns of this table, we remove this flag so it will not be analyzed again.
This pr (#49918) skip collecting value columns of agg and unique mor table, but didn't change the logic to remove the new partition flag. The old logic require all columns has been collected before remove the flag, but value columns are skipped, so the flag will never be removed. This will cause the table be analyzed repeatedly. This pr is to fix the bug.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-05-21T07:47:33Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Jibing-Li · 2025-05-21T07:49:41Z

run buildall

doris-robot · 2025-05-21T08:08:55Z

TPC-H: Total hot run time: 33699 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 038077918b3d67d39b537d204b0683877bb30bad, data reload: false

------ Round 1 ----------------------------------
q1	25991	5020	4945	4945
q2	2085	297	194	194
q3	10363	1241	677	677
q4	10221	1005	534	534
q5	7557	2410	2271	2271
q6	182	165	131	131
q7	931	752	617	617
q8	9312	1276	1094	1094
q9	6790	5043	5106	5043
q10	6811	2292	1910	1910
q11	480	281	277	277
q12	353	347	223	223
q13	17792	3636	3100	3100
q14	238	234	218	218
q15	533	486	494	486
q16	406	441	369	369
q17	604	842	358	358
q18	7500	7213	7204	7204
q19	1255	934	519	519
q20	340	335	224	224
q21	3732	2605	2356	2356
q22	1045	1014	949	949
Total cold run time: 114521 ms
Total hot run time: 33699 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5041	5016	5032	5016
q2	240	331	230	230
q3	2121	2610	2283	2283
q4	1367	1753	1379	1379
q5	4289	4309	4407	4309
q6	217	173	125	125
q7	1975	1923	1787	1787
q8	2586	2648	2568	2568
q9	7161	7230	7051	7051
q10	3003	3137	2762	2762
q11	582	538	495	495
q12	683	762	620	620
q13	3476	3857	3233	3233
q14	272	305	278	278
q15	511	477	474	474
q16	459	493	456	456
q17	1149	1589	1352	1352
q18	7781	7574	7444	7444
q19	795	756	937	756
q20	1967	2038	1896	1896
q21	4798	4451	4440	4440
q22	1078	1053	1000	1000
Total cold run time: 51551 ms
Total hot run time: 49954 ms

doris-robot · 2025-05-21T08:21:10Z

TPC-DS: Total hot run time: 193646 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 038077918b3d67d39b537d204b0683877bb30bad, data reload: false

query1	1438	1096	1078	1078
query2	6201	1818	1827	1818
query3	10998	4550	4472	4472
query4	53261	25460	23769	23769
query5	5255	503	447	447
query6	335	194	183	183
query7	4936	512	295	295
query8	313	246	224	224
query9	5909	2635	2641	2635
query10	436	338	269	269
query11	15087	15044	15062	15044
query12	166	117	109	109
query13	1065	523	424	424
query14	10081	6231	6297	6231
query15	211	221	199	199
query16	6996	655	537	537
query17	1113	731	612	612
query18	1573	416	328	328
query19	204	197	194	194
query20	136	120	115	115
query21	208	127	117	117
query22	4485	4455	4374	4374
query23	34257	33548	33460	33460
query24	6892	2397	2459	2397
query25	463	467	418	418
query26	752	274	161	161
query27	2451	518	353	353
query28	3454	2155	2146	2146
query29	586	554	446	446
query30	272	216	188	188
query31	854	880	819	819
query32	73	65	64	64
query33	478	374	304	304
query34	790	846	559	559
query35	793	856	732	732
query36	933	992	894	894
query37	115	98	76	76
query38	4232	4282	4169	4169
query39	1636	1454	1461	1454
query40	214	126	108	108
query41	56	54	52	52
query42	127	121	110	110
query43	506	516	478	478
query44	1301	849	853	849
query45	183	178	169	169
query46	866	1038	672	672
query47	1845	1861	1814	1814
query48	398	439	333	333
query49	708	500	457	457
query50	652	680	413	413
query51	4332	4289	4293	4289
query52	114	111	101	101
query53	225	261	189	189
query54	597	591	526	526
query55	91	85	88	85
query56	336	294	306	294
query57	1253	1194	1136	1136
query58	275	277	260	260
query59	2714	2788	2605	2605
query60	345	328	318	318
query61	140	123	124	123
query62	737	757	734	734
query63	223	188	198	188
query64	2302	1004	723	723
query65	4323	4246	4234	4234
query66	753	399	306	306
query67	15993	15524	15464	15464
query68	6954	885	513	513
query69	540	310	266	266
query70	1137	1121	1108	1108
query71	491	320	301	301
query72	5965	4739	4864	4739
query73	1369	646	361	361
query74	8894	9147	8983	8983
query75	3885	3178	2699	2699
query76	4199	1191	744	744
query77	613	366	286	286
query78	10198	10162	9295	9295
query79	2476	784	584	584
query80	608	507	499	499
query81	485	261	230	230
query82	436	122	99	99
query83	411	250	234	234
query84	295	112	83	83
query85	787	351	314	314
query86	379	314	286	286
query87	4417	4511	4307	4307
query88	3609	2333	2326	2326
query89	406	326	287	287
query90	1786	218	210	210
query91	144	141	117	117
query92	78	62	57	57
query93	1898	932	578	578
query94	667	416	305	305
query95	377	299	286	286
query96	518	573	286	286
query97	2761	2767	2625	2625
query98	253	220	208	208
query99	1474	1427	1286	1286
Total cold run time: 299557 ms
Total hot run time: 193646 ms

doris-robot · 2025-05-21T08:26:18Z

ClickBench: Total hot run time: 29.02 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 038077918b3d67d39b537d204b0683877bb30bad, data reload: false

query1	0.04	0.03	0.03
query2	0.13	0.10	0.11
query3	0.25	0.19	0.20
query4	1.60	0.20	0.19
query5	0.48	0.45	0.45
query6	1.15	0.67	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.59	0.51	0.52
query10	0.56	0.58	0.57
query11	0.16	0.12	0.11
query12	0.14	0.11	0.11
query13	0.60	0.60	0.59
query14	0.79	0.81	0.80
query15	0.88	0.89	0.85
query16	0.38	0.41	0.37
query17	1.06	1.02	1.06
query18	0.23	0.21	0.21
query19	1.94	1.83	1.78
query20	0.01	0.02	0.01
query21	15.46	0.89	0.54
query22	0.76	1.31	0.69
query23	14.76	1.40	0.62
query24	6.96	1.06	0.77
query25	0.48	0.09	0.12
query26	0.64	0.16	0.16
query27	0.06	0.05	0.05
query28	9.14	0.94	0.43
query29	12.56	4.05	3.35
query30	0.26	0.09	0.06
query31	2.83	0.58	0.38
query32	3.23	0.54	0.48
query33	3.09	3.12	3.04
query34	15.68	5.09	4.47
query35	4.51	4.53	4.50
query36	0.66	0.51	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.03
query40	0.17	0.13	0.12
query41	0.09	0.03	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 102.62 s
Total hot run time: 29.02 s

fe/fe-core/src/main/java/org/apache/doris/statistics/TableStatsMeta.java

Jibing-Li · 2025-05-22T11:35:25Z

run buildall

doris-robot · 2025-05-22T11:55:08Z

TPC-H: Total hot run time: 34078 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e7c18b585e936104fde636f05f720488c4ee7937, data reload: false

------ Round 1 ----------------------------------
q1	26049	5097	5052	5052
q2	2067	282	184	184
q3	10441	1258	720	720
q4	10236	1001	535	535
q5	7675	2444	2318	2318
q6	190	174	137	137
q7	924	759	606	606
q8	9323	1300	1143	1143
q9	6792	5149	5116	5116
q10	6817	2310	1896	1896
q11	480	293	274	274
q12	349	353	214	214
q13	17773	3677	3143	3143
q14	234	224	213	213
q15	548	510	506	506
q16	409	427	365	365
q17	590	858	361	361
q18	7831	7183	7087	7087
q19	1212	930	570	570
q20	340	329	223	223
q21	3920	3206	2442	2442
q22	1062	1022	973	973
Total cold run time: 115262 ms
Total hot run time: 34078 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5467	5066	5305	5066
q2	247	333	228	228
q3	2186	2646	2264	2264
q4	1385	1800	1411	1411
q5	4576	4467	4399	4399
q6	211	168	129	129
q7	1966	1894	1743	1743
q8	2634	2538	2507	2507
q9	7197	7150	7344	7150
q10	2980	3191	2750	2750
q11	594	522	509	509
q12	697	780	658	658
q13	3525	3890	3282	3282
q14	280	301	260	260
q15	551	483	494	483
q16	433	479	446	446
q17	1164	1588	1354	1354
q18	7590	7605	7559	7559
q19	799	910	978	910
q20	1969	2043	1908	1908
q21	4763	4261	4349	4261
q22	1088	1023	989	989
Total cold run time: 52302 ms
Total hot run time: 50266 ms

doris-robot · 2025-05-22T12:06:42Z

TPC-DS: Total hot run time: 185761 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e7c18b585e936104fde636f05f720488c4ee7937, data reload: false

query1	1026	474	505	474
query2	6567	1855	1820	1820
query3	6738	238	217	217
query4	26084	23793	23125	23125
query5	4315	602	472	472
query6	326	215	197	197
query7	4624	497	284	284
query8	293	250	240	240
query9	8601	2613	2616	2613
query10	497	323	274	274
query11	15645	15116	14917	14917
query12	158	114	108	108
query13	1664	537	404	404
query14	9433	6383	6407	6383
query15	213	190	170	170
query16	7297	642	467	467
query17	1191	721	587	587
query18	1981	396	316	316
query19	197	192	163	163
query20	123	123	123	123
query21	213	126	113	113
query22	4116	4141	4028	4028
query23	33896	33065	33075	33065
query24	8452	2366	2387	2366
query25	535	483	394	394
query26	1228	265	150	150
query27	2747	492	339	339
query28	4353	2134	2116	2116
query29	749	544	447	447
query30	278	219	187	187
query31	942	866	751	751
query32	78	67	65	65
query33	562	406	313	313
query34	795	851	505	505
query35	799	815	724	724
query36	937	976	902	902
query37	112	102	77	77
query38	4058	4123	4075	4075
query39	1458	1407	1421	1407
query40	211	118	104	104
query41	59	51	52	51
query42	122	110	104	104
query43	494	499	467	467
query44	1301	809	810	809
query45	179	174	167	167
query46	834	1018	653	653
query47	1761	1808	1731	1731
query48	394	419	305	305
query49	754	509	428	428
query50	654	689	395	395
query51	4215	4126	4091	4091
query52	104	106	95	95
query53	217	251	186	186
query54	585	567	513	513
query55	84	80	81	80
query56	296	298	300	298
query57	1172	1152	1115	1115
query58	260	253	248	248
query59	2622	2714	2622	2622
query60	329	320	307	307
query61	125	127	124	124
query62	805	699	652	652
query63	225	185	188	185
query64	4339	996	659	659
query65	4359	4213	4214	4213
query66	1137	407	315	315
query67	15858	15679	15303	15303
query68	8269	877	515	515
query69	487	308	264	264
query70	1242	1147	1066	1066
query71	494	326	302	302
query72	5631	4693	4665	4665
query73	693	593	345	345
query74	9334	9071	8817	8817
query75	3927	3188	2686	2686
query76	3755	1202	739	739
query77	791	361	286	286
query78	10004	10099	9297	9297
query79	2896	819	584	584
query80	636	519	442	442
query81	469	263	220	220
query82	464	127	95	95
query83	284	249	233	233
query84	283	105	91	91
query85	797	354	313	313
query86	339	319	296	296
query87	4488	4435	4305	4305
query88	3258	2381	2315	2315
query89	419	321	292	292
query90	1957	229	214	214
query91	153	137	114	114
query92	76	62	56	56
query93	1601	942	581	581
query94	673	426	306	306
query95	381	301	286	286
query96	502	571	287	287
query97	2704	2762	2656	2656
query98	230	213	210	210
query99	1461	1454	1285	1285
Total cold run time: 275474 ms
Total hot run time: 185761 ms

doris-robot · 2025-05-22T12:11:51Z

ClickBench: Total hot run time: 28.56 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e7c18b585e936104fde636f05f720488c4ee7937, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.11	0.10
query3	0.24	0.20	0.19
query4	1.60	0.18	0.18
query5	0.51	0.45	0.44
query6	1.58	0.65	0.66
query7	0.02	0.02	0.01
query8	0.04	0.04	0.04
query9	0.58	0.51	0.50
query10	0.58	0.56	0.56
query11	0.15	0.11	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.61
query14	0.79	0.81	0.80
query15	0.86	0.86	0.85
query16	0.38	0.38	0.38
query17	1.00	0.99	1.06
query18	0.22	0.21	0.20
query19	1.93	1.81	1.79
query20	0.01	0.01	0.01
query21	15.47	0.90	0.54
query22	0.75	1.06	0.69
query23	15.04	1.37	0.66
query24	7.88	0.93	0.30
query25	0.42	0.15	0.12
query26	0.64	0.17	0.15
query27	0.06	0.05	0.04
query28	9.26	0.92	0.46
query29	12.57	3.95	3.30
query30	0.25	0.09	0.07
query31	2.82	0.60	0.39
query32	3.23	0.56	0.47
query33	3.01	3.05	3.11
query34	15.83	5.03	4.49
query35	4.47	4.50	4.51
query36	0.66	0.49	0.47
query37	0.08	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.15	0.13
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 104.24 s
Total hot run time: 28.56 s

Jibing-Li · 2025-05-22T15:31:38Z

run p0

github-actions · 2025-05-23T02:53:32Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-05-23T02:53:34Z

PR approved by anyone and no changes requested.

zfr9527

LGTM

…partition flag bug. (apache#51116) ### What problem does this PR solve? Related PR: apache#49918 Problem Summary: Fix sample analyze agg/mor table doesn't remove new partition flag bug. When a new partition load data for the first time, we set a flag to this table's metadata, this flag indicate that the statistics data of this table is expired and not accurate, so it will trigger a new collection for this table. After collect all columns of this table, we remove this flag so it will not be analyzed again. This pr (apache#49918) skip collecting value columns of agg and unique mor table, but didn't change the logic to remove the new partition flag. The old logic require all columns has been collected before remove the flag, but value columns are skipped, so the flag will never be removed. This will cause the table be analyzed repeatedly. This pr is to fix the bug.

Jibing-Li marked this pull request as ready for review May 21, 2025 07:50

morrySnow reviewed May 22, 2025

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/statistics/TableStatsMeta.java Outdated Show resolved Hide resolved

Fix sample analyze agg/mor table doesn't remove new partition flag bug.

e7c18b5

Jibing-Li force-pushed the fixautoanalyze branch from 0380779 to e7c18b5 Compare May 22, 2025 11:34

morrySnow approved these changes May 23, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label May 23, 2025

github-actions bot added the reviewed label May 23, 2025

zfr9527 approved these changes May 23, 2025

View reviewed changes

Jibing-Li merged commit ea62d6b into apache:master May 23, 2025
27 of 29 checks passed

Jibing-Li deleted the fixautoanalyze branch May 23, 2025 04:14

Jibing-Li added not-merge/2.1 not-merge/3.0 labels Jun 12, 2025

Jibing-Li mentioned this pull request Jun 23, 2025

branch-3.0:[improvement](statistics)Agg table set preagg on when doing sample analyzing. (#49918) #51672

Merged

Jibing-Li mentioned this pull request Jun 23, 2025

branch-2.1: [improvement](statistics)Agg table set preagg on when doing sample analyzing. (#49918) #51675

Merged

Jibing-Li added dev/3.1.0-merged dev/2.1.11-merged dev/3.0.7-merged and removed not-merge/2.1 not-merge/3.0 labels Jun 23, 2025

[fix](statistics)Fix sample analyze agg/mor table doesn't remove new partition flag bug. #51116

[fix](statistics)Fix sample analyze agg/mor table doesn't remove new partition flag bug. #51116

Uh oh!

Conversation

Jibing-Li commented May 21, 2025 • edited by morrySnow Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented May 21, 2025

Uh oh!

Jibing-Li commented May 21, 2025

Uh oh!

doris-robot commented May 21, 2025

Uh oh!

doris-robot commented May 21, 2025

Uh oh!

doris-robot commented May 21, 2025

Uh oh!

Uh oh!

Jibing-Li commented May 22, 2025

Uh oh!

doris-robot commented May 22, 2025

Uh oh!

doris-robot commented May 22, 2025

Uh oh!

doris-robot commented May 22, 2025

Uh oh!

Jibing-Li commented May 22, 2025

Uh oh!

github-actions bot commented May 23, 2025

Uh oh!

github-actions bot commented May 23, 2025

Uh oh!

zfr9527 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Jibing-Li commented May 21, 2025 •

edited by morrySnow

Loading