[Perf up to -44%] System.Globalization.Tests.StringHash regression on Windows due to change to ICU

[DrewScoggins]

Run Information

Architecture	x64
OS	Windows 10.0.18362
Changes	diff

Regressions in System.Globalization.Tests.StringHash

Benchmark	Baseline	Test	Test/Base	Modality	Baseline Outlier
GetHashCode	2.04 μs	2.57 μs	1.26		True
GetHashCode	2.04 μs	2.56 μs	1.26		False

Historical Data in Reporting System

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f netcoreapp5.0 --filter 'System.Globalization.Tests.StringHash*';

Details

Histogram

System.Globalization.Tests.StringHash.GetHashCode(Count: 128, Options: (en-US, None))

[1990.277 ; 2055.859) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[2055.859 ; 2128.177) | @@@@@@
[2128.177 ; 2216.432) | @@
[2216.432 ; 2282.014) | 
[2282.014 ; 2347.595) | 
[2347.595 ; 2413.177) | 
[2413.177 ; 2478.759) | 
[2478.759 ; 2556.059) | @@
[2556.059 ; 2621.641) | @@@@@@@@@@@@@@@@@@

System.Globalization.Tests.StringHash.GetHashCode(Count: 128, Options: (, None))

[1982.547 ; 2049.339) | @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[2049.339 ; 2100.801) | @@@@@@
[2100.801 ; 2172.655) | @
[2172.655 ; 2239.447) | @@
[2239.447 ; 2306.240) | 
[2306.240 ; 2373.032) | 
[2373.032 ; 2439.824) | 
[2439.824 ; 2544.466) | 
[2544.466 ; 2660.679) | @@@@@@@@@@@@@@@@@@@@

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Tagging subscribers to this area: @tarekgh, @safern, @krwq
Notify danmosemsft if you want to be subscribed.

[DrewScoggins]

Suspect this is from the ICU change @adamsitnik

[danmoseley]

38c2d55...5395b14 which indeed straddles ICU change 1db45ed

[danmoseley]

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.421 (2004/?/20H1)
Intel Core i7-7700 CPU 3.60GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=5.0.100-rc.1.20401.3
[Host] : .NET Core 5.0.0 (CoreCLR 5.0.20.40101, CoreFX 5.0.20.40101), X64 RyuJIT
Job-OKFWGW : .NET Core 3.1.6 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.31603), X64 RyuJIT
Job-JENQMI : .NET Core 5.0.0 (CoreCLR 5.0.20.40101, CoreFX 5.0.20.40101), X64 RyuJIT

PowerPlanMode=00000000-0000-0000-0000-000000000000 Arguments=/p:DebugType=portable IterationTime=250.0000 ms
MaxIterationCount=20 MinIterationCount=15 WarmupCount=1

Method	Job	Runtime	Toolchain	Count	Options	Mean	Error	StdDev	Median	Min	Max	Ratio	RatioSD	Gen 0	Gen 1	Gen 2	Allocated
GetHashCode	Job-OKFWGW	.NET Core 3.1	netcoreapp3.1	128	(, IgnoreCase)	1,621.08 ns	19.271 ns	17.083 ns	1,618.77 ns	1,592.22 ns	1,647.63 ns	1.00	0.00	-	-	-	-
GetHashCode	Job-JENQMI	.NET Core 5.0	netcoreapp5.0	128	(, IgnoreCase)	1,805.31 ns	25.102 ns	22.252 ns	1,803.28 ns	1,773.23 ns	1,852.56 ns	1.11	0.02	-	-	-	-
GetHashCode	Job-OKFWGW	.NET Core 3.1	netcoreapp3.1	128	(, None)	1,597.57 ns	12.449 ns	11.644 ns	1,597.08 ns	1,578.13 ns	1,619.33 ns	1.00	0.00	-	-	-	-
GetHashCode	Job-JENQMI	.NET Core 5.0	netcoreapp5.0	128	(, None)	2,292.88 ns	11.368 ns	9.493 ns	2,295.09 ns	2,278.58 ns	2,313.61 ns	1.44	0.01	-	-	-	-
GetHashCode	Job-OKFWGW	.NET Core 3.1	netcoreapp3.1	128	(en-US, IgnoreCase)	1,609.98 ns	18.390 ns	16.302 ns	1,612.23 ns	1,577.50 ns	1,635.30 ns	1.00	0.00	-	-	-	-
GetHashCode	Job-JENQMI	.NET Core 5.0	netcoreapp5.0	128	(en-US, IgnoreCase)	1,802.87 ns	9.473 ns	7.396 ns	1,802.87 ns	1,788.38 ns	1,813.72 ns	1.12	0.01	-	-	-	-
GetHashCode	Job-OKFWGW	.NET Core 3.1	netcoreapp3.1	128	(en-US, None)	1,597.00 ns	6.769 ns	6.000 ns	1,596.01 ns	1,585.12 ns	1,606.54 ns	1.00	0.00	-	-	-	-
GetHashCode	Job-JENQMI	.NET Core 5.0	netcoreapp5.0	128	(en-US, None)	2,297.21 ns	14.340 ns	12.712 ns	2,298.81 ns	2,273.96 ns	2,326.05 ns	1.44	0.01	-	-	-	-
GetHashCode	Job-OKFWGW	.NET Core 3.1	netcoreapp3.1	128	(en-US, Ordinal)	94.29 ns	0.687 ns	0.642 ns	94.27 ns	93.17 ns	95.24 ns	1.00	0.00	-	-	-	-
GetHashCode	Job-JENQMI	.NET Core 5.0	netcoreapp5.0	128	(en-US, Ordinal)	94.41 ns	0.693 ns	0.648 ns	94.46 ns	93.22 ns	95.42 ns	1.00	0.01	-	-	-	-
GetHashCode	Job-OKFWGW	.NET Core 3.1	netcoreapp3.1	128	(en-US, OrdinalIgnoreCase)	109.47 ns	0.790 ns	0.700 ns	109.60 ns	107.92 ns	110.50 ns	1.00	0.00	-	-	-	-
GetHashCode	Job-JENQMI	.NET Core 5.0	netcoreapp5.0	128	(en-US, OrdinalIgnoreCase)	104.76 ns	0.944 ns	0.789 ns	104.64 ns	103.68 ns	106.49 ns	0.96	0.01	-	-	-	-
GetHashCode	Job-OKFWGW	.NET Core 3.1	netcoreapp3.1	131072	(, IgnoreCase)	2,071,652.88 ns	22,931.688 ns	19,148.998 ns	2,066,778.12 ns	2,042,522.66 ns	2,116,059.38 ns	1.00	0.00	-	-	-	-
GetHashCode	Job-JENQMI	.NET Core 5.0	netcoreapp5.0	131072	(, IgnoreCase)	1,718,811.56 ns	19,385.193 ns	17,184.469 ns	1,713,651.25 ns	1,691,481.25 ns	1,751,724.38 ns	0.83	0.01	-	-	-	2 B
GetHashCode	Job-OKFWGW	.NET Core 3.1	netcoreapp3.1	131072	(, None)	2,048,322.37 ns	13,048.152 ns	12,205.249 ns	2,045,656.64 ns	2,030,086.33 ns	2,074,140.23 ns	1.00	0.00	-	-	-	-
GetHashCode	Job-JENQMI	.NET Core 5.0	netcoreapp5.0	131072	(, None)	1,843,248.61 ns	14,119.257 ns	13,207.162 ns	1,839,734.72 ns	1,827,571.53 ns	1,866,947.92 ns	0.90	0.01	-	-	-	2 B
GetHashCode	Job-OKFWGW	.NET Core 3.1	netcoreapp3.1	131072	(en-US, IgnoreCase)	2,084,957.37 ns	12,521.864 ns	11,100.306 ns	2,085,180.08 ns	2,067,164.06 ns	2,109,292.19 ns	1.00	0.00	-	-	-	-
GetHashCode	Job-JENQMI	.NET Core 5.0	netcoreapp5.0	131072	(en-US, IgnoreCase)	1,720,143.70 ns	10,942.028 ns	9,699.823 ns	1,718,112.15 ns	1,704,280.56 ns	1,741,363.19 ns	0.83	0.01	-	-	-	2 B
GetHashCode	Job-OKFWGW	.NET Core 3.1	netcoreapp3.1	131072	(en-US, None)	2,060,049.44 ns	16,861.825 ns	14,947.569 ns	2,056,449.22 ns	2,039,420.31 ns	2,088,351.56 ns	1.00	0.00	-	-	-	-
GetHashCode	Job-JENQMI	.NET Core 5.0	netcoreapp5.0	131072	(en-US, None)	1,832,617.95 ns	10,946.901 ns	9,141.158 ns	1,833,052.78 ns	1,815,781.25 ns	1,847,127.08 ns	0.89	0.01	-	-	-	2 B
GetHashCode	Job-OKFWGW	.NET Core 3.1	netcoreapp3.1	131072	(en-US, Ordinal)	92,748.93 ns	411.805 ns	365.055 ns	92,735.15 ns	91,903.55 ns	93,411.09 ns	1.00	0.00	-	-	-	-
GetHashCode	Job-JENQMI	.NET Core 5.0	netcoreapp5.0	131072	(en-US, Ordinal)	92,354.27 ns	536.458 ns	475.556 ns	92,275.22 ns	91,708.41 ns	93,192.87 ns	1.00	0.01	-	-	-	-
GetHashCode	Job-OKFWGW	.NET Core 3.1	netcoreapp3.1	131072	(en-US, OrdinalIgnoreCase)	99,808.81 ns	801.945 ns	750.140 ns	99,875.16 ns	98,514.53 ns	101,361.15 ns	1.00	0.00	-	-	-	-
GetHashCode	Job-JENQMI	.NET Core 5.0	netcoreapp5.0	131072	(en-US, OrdinalIgnoreCase)	100,253.98 ns	1,948.712 ns	1,627.262 ns	100,225.28 ns	98,437.18 ns	104,460.55 ns	1.00	0.02	-	-	-	-

with set DOTNET_SYSTEM_GLOBALIZATION_USENLS=1 the regression vanishes

Method	Job	Runtime	Toolchain	Count	Options	Mean	Error	StdDev	Median	Min	Max	Ratio	RatioSD	Gen 0	Gen 1	Gen 2	Allocated
GetHashCode	Job-HXZHYN	.NET Core 3.1	netcoreapp3.1	128	(, IgnoreCase)	1,629.44 ns	12.351 ns	10.313 ns	1,626.29 ns	1,612.77 ns	1,647.74 ns	1.00	0.00	-	-	-	-
GetHashCode	Job-RTNHQJ	.NET Core 5.0	netcoreapp5.0	128	(, IgnoreCase)	1,639.70 ns	16.098 ns	15.058 ns	1,641.33 ns	1,616.31 ns	1,663.52 ns	1.01	0.01	-	-	-	-
GetHashCode	Job-HXZHYN	.NET Core 3.1	netcoreapp3.1	128	(, None)	1,600.14 ns	12.896 ns	12.063 ns	1,596.47 ns	1,577.67 ns	1,622.99 ns	1.00	0.00	-	-	-	-
GetHashCode	Job-RTNHQJ	.NET Core 5.0	netcoreapp5.0	128	(, None)	1,617.61 ns	10.560 ns	9.878 ns	1,616.29 ns	1,601.19 ns	1,641.91 ns	1.01	0.01	-	-	-	-
GetHashCode	Job-HXZHYN	.NET Core 3.1	netcoreapp3.1	128	(en-US, IgnoreCase)	1,634.28 ns	15.925 ns	14.897 ns	1,632.31 ns	1,611.36 ns	1,659.83 ns	1.00	0.00	-	-	-	-
GetHashCode	Job-RTNHQJ	.NET Core 5.0	netcoreapp5.0	128	(en-US, IgnoreCase)	1,628.90 ns	12.031 ns	10.046 ns	1,631.88 ns	1,612.01 ns	1,647.96 ns	1.00	0.01	-	-	-	-
GetHashCode	Job-HXZHYN	.NET Core 3.1	netcoreapp3.1	128	(en-US, None)	1,609.62 ns	13.546 ns	12.671 ns	1,611.68 ns	1,588.45 ns	1,632.00 ns	1.00	0.00	-	-	-	-
GetHashCode	Job-RTNHQJ	.NET Core 5.0	netcoreapp5.0	128	(en-US, None)	1,625.32 ns	17.832 ns	14.891 ns	1,621.75 ns	1,606.32 ns	1,662.65 ns	1.01	0.01	-	-	-	-
GetHashCode	Job-HXZHYN	.NET Core 3.1	netcoreapp3.1	128	(en-US, Ordinal)	94.45 ns	0.496 ns	0.440 ns	94.44 ns	93.34 ns	95.01 ns	1.00	0.00	-	-	-	-
GetHashCode	Job-RTNHQJ	.NET Core 5.0	netcoreapp5.0	128	(en-US, Ordinal)	94.33 ns	0.483 ns	0.452 ns	94.41 ns	93.41 ns	94.90 ns	1.00	0.01	-	-	-	-
GetHashCode	Job-HXZHYN	.NET Core 3.1	netcoreapp3.1	128	(en-US, OrdinalIgnoreCase)	108.02 ns	0.624 ns	0.583 ns	108.00 ns	106.91 ns	109.15 ns	1.00	0.00	-	-	-	-
GetHashCode	Job-RTNHQJ	.NET Core 5.0	netcoreapp5.0	128	(en-US, OrdinalIgnoreCase)	103.97 ns	0.973 ns	0.812 ns	103.70 ns	102.40 ns	105.33 ns	0.96	0.01	-	-	-	-
GetHashCode	Job-HXZHYN	.NET Core 3.1	netcoreapp3.1	131072	(, IgnoreCase)	2,075,352.60 ns	23,314.712 ns	21,808.596 ns	2,073,708.59 ns	2,045,172.66 ns	2,115,289.84 ns	1.00	0.00	-	-	-	-
GetHashCode	Job-RTNHQJ	.NET Core 5.0	netcoreapp5.0	131072	(, IgnoreCase)	2,055,916.29 ns	11,188.943 ns	9,343.275 ns	2,056,950.00 ns	2,041,283.59 ns	2,075,545.31 ns	0.99	0.01	-	-	-	2 B
GetHashCode	Job-HXZHYN	.NET Core 3.1	netcoreapp3.1	131072	(, None)	2,041,808.93 ns	19,653.356 ns	17,422.188 ns	2,037,607.03 ns	2,015,582.03 ns	2,076,410.16 ns	1.00	0.00	-	-	-	-
GetHashCode	Job-RTNHQJ	.NET Core 5.0	netcoreapp5.0	131072	(, None)	2,083,193.12 ns	24,187.443 ns	22,624.950 ns	2,077,690.62 ns	2,057,233.59 ns	2,138,438.28 ns	1.02	0.02	-	-	-	2 B
GetHashCode	Job-HXZHYN	.NET Core 3.1	netcoreapp3.1	131072	(en-US, IgnoreCase)	2,065,399.61 ns	12,895.803 ns	11,431.793 ns	2,063,775.78 ns	2,043,192.19 ns	2,086,204.69 ns	1.00	0.00	-	-	-	-
GetHashCode	Job-RTNHQJ	.NET Core 5.0	netcoreapp5.0	131072	(en-US, IgnoreCase)	2,141,078.83 ns	74,478.308 ns	85,769.330 ns	2,112,928.12 ns	2,050,237.50 ns	2,302,097.66 ns	1.03	0.04	-	-	-	4 B
GetHashCode	Job-HXZHYN	.NET Core 3.1	netcoreapp3.1	131072	(en-US, None)	2,041,968.02 ns	15,368.610 ns	13,623.872 ns	2,041,774.22 ns	2,025,425.00 ns	2,073,412.50 ns	1.00	0.00	-	-	-	-
GetHashCode	Job-RTNHQJ	.NET Core 5.0	netcoreapp5.0	131072	(en-US, None)	2,058,715.51 ns	16,039.631 ns	14,218.715 ns	2,057,590.23 ns	2,041,064.06 ns	2,093,275.00 ns	1.01	0.01	-	-	-	2 B
GetHashCode	Job-HXZHYN	.NET Core 3.1	netcoreapp3.1	131072	(en-US, Ordinal)	93,103.63 ns	646.697 ns	540.022 ns	93,292.66 ns	92,387.13 ns	94,005.13 ns	1.00	0.00	-	-	-	-
GetHashCode	Job-RTNHQJ	.NET Core 5.0	netcoreapp5.0	131072	(en-US, Ordinal)	92,167.94 ns	349.044 ns	291.468 ns	92,165.02 ns	91,647.59 ns	92,807.42 ns	0.99	0.01	-	-	-	-
GetHashCode	Job-HXZHYN	.NET Core 3.1	netcoreapp3.1	131072	(en-US, OrdinalIgnoreCase)	99,660.67 ns	753.598 ns	668.045 ns	99,806.33 ns	98,058.64 ns	100,611.35 ns	1.00	0.00	-	-	-	-
GetHashCode	Job-RTNHQJ	.NET Core 5.0	netcoreapp5.0	131072	(en-US, OrdinalIgnoreCase)	99,506.65 ns	850.553 ns	753.993 ns	99,726.39 ns	98,075.20 ns	100,561.46 ns	1.00	0.01	-	-	-	-

Note the regression is largest for small sizes, case sensitive. Does it all end up in IcuGetHashCodeOfString ?

[danmoseley]

I guess @adamsitnik already looked carefully at that in dotnet/coreclr#24889.

Might be interesting to verify Windows vs. Linux perf. @DrewScoggins how can I pull that comparison?