[Perf] Linux/arm64: 175 Regressions on 4/12/2023 2:59:13 PM

[performanceautofiler]

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	7a57f6dd800e77508a4ba405e623059935c82f43
Compare	887c043eb94be364188e2b23a87efa214ea57f1e
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Numerics.Tests.Perf_Vector3

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
EqualsBenchmark - Duration of single invocation	7.94 ns	26.91 ns	3.39	0.17	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

• Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
• CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
• AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
• Dotnet SDK installed for dotnet commands
• Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_Vector3*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_Vector3* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Numerics.Tests.Perf_Vector3*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Numerics.Tests.Perf_Vector3* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Details

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_Vector3.EqualsBenchmark

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 26.914583093396324 > 8.55337526931995.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -261.11484907013727 (T) = (0 -27.006181263165292) / Math.Sqrt((0.03418039350941472 / (19)) + (0.017285607105997216 / (5))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (5) - 2, .025) and -2.3441521386437416 = (8.07564373375607 - 27.006181263165292) / 8.07564373375607 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[performanceautofiler]

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	7a57f6dd800e77508a4ba405e623059935c82f43
Compare	887c043eb94be364188e2b23a87efa214ea57f1e
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
MatchWord - Duration of single invocation	572.29 ns	1.29 μs	2.25	0.03	True
MatchWord - Duration of single invocation	769.82 ns	1.74 μs	2.27	0.01	True
Uri_IsMatch - Duration of single invocation	416.41 ns	489.89 ns	1.18	0.03	False
MatchesWords - Duration of single invocation	170.41 μs	230.69 μs	1.35	0.02	False
OneNodeBacktracking - Duration of single invocation	430.64 ns	1.08 μs	2.50	0.02	False
IP_IsMatch - Duration of single invocation	484.90 ns	750.18 ns	1.55	0.01	True
SplitWords - Duration of single invocation	21.79 μs	39.76 μs	1.82	0.01	False
IP_IsNotMatch - Duration of single invocation	489.11 ns	761.45 ns	1.56	0.02	True
ReplaceWords - Duration of single invocation	17.18 μs	30.13 μs	1.75	0.02	False
MatchesWords - Duration of single invocation	14.99 μs	27.94 μs	1.86	0.02	True
MatchWord - Duration of single invocation	4.83 μs	6.53 μs	1.35	0.01	False
MatchesWords - Duration of single invocation	20.33 μs	38.53 μs	1.89	0.01	True
SplitWords - Duration of single invocation	171.48 μs	232.70 μs	1.36	0.01	False
SplitWords - Duration of single invocation	16.39 μs	29.56 μs	1.80	0.02	False
Uri_IsNotMatch - Duration of single invocation	557.39 ns	741.12 ns	1.33	0.03	False
ReplaceWords - Duration of single invocation	172.42 μs	231.49 μs	1.34	0.02	False
Backtracking - Duration of single invocation	551.30 ns	1.01 μs	1.83	0.01	True
IP_IsMatch - Duration of single invocation	497.12 ns	761.00 ns	1.53	0.02	True
OneNodeBacktracking - Duration of single invocation	341.43 ns	778.28 ns	2.28	0.02	True
Uri_IsNotMatch - Duration of single invocation	551.08 ns	755.83 ns	1.37	0.02	False
Backtracking - Duration of single invocation	625.83 ns	800.77 ns	1.28	0.04	False
ReplaceWords - Duration of single invocation	22.52 μs	40.74 μs	1.81	0.01	False
IP_IsNotMatch - Duration of single invocation	484.26 ns	757.26 ns	1.56	0.01	True
MatchesWord - Duration of single invocation	10.13 μs	20.67 μs	2.04	0.01	True
Uri_IsMatch - Duration of single invocation	405.04 ns	493.14 ns	1.22	0.02	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

• Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
• CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
• AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
• Dotnet SDK installed for dotnet commands
• Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Text.RegularExpressions.Tests.Perf_Regex_Common* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Text.RegularExpressions.Tests.Perf_Regex_Common* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Details

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchWord(Options: Compiled)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2860160637332179 > 607.3851504068558.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -173.09519787035202 (T) = (0 -1292.399964157456) / Math.Sqrt((25.945980539833812 / (20)) + (62.65837485702075 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -1.2301578475337775 = (579.5105335645447 - 1292.399964157456) / 579.5105335645447 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchWord(Options: IgnoreCase, Compiled)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.7446261578847875 > 804.4021069560849.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -198.66096816720977 (T) = (0 -1740.2209896308277) / Math.Sqrt((41.5437687672291 / (20)) + (87.95711664434917 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -1.2729024150090862 = (765.638233361581 - 1740.2209896308277) / 765.638233361581 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsMatch(Options: IgnoreCase, Compiled)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 489.8902451638566 > 431.8049071607993.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -24.404658280640415 (T) = (0 -497.61027834537197) / Math.Sqrt((11.525195906940164 / (20)) + (49.43988774729283 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.21417487596906684 = (409.83410890314735 - 497.61027834537197) / 409.83410890314735 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWords(Options: None)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 230.68647549019607 > 179.51691233865492.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -103.64675132648438 (T) = (0 -231269.79991515836) / Math.Sqrt((129295.42519646467 / (20)) + (1334378.2327768167 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.3538119089614285 = (170828.60505531827 - 231269.79991515836) / 170828.60505531827 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Common.OneNodeBacktracking(Options: IgnoreCase, Compiled)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.0782254376292215 > 452.20639815281334.
IsChangePoint: Marked as a change because one of 3/28/2023 2:42:21 AM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -143.02957795749495 (T) = (0 -1071.907792252583) / Math.Sqrt((28.840270665640173 / (20)) + (55.902441371388484 / (3))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (3) - 2, .025) and -1.486775026674144 = (431.0433315257195 - 1071.907792252583) / 431.0433315257195 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Common.IP_IsMatch(Options: Compiled)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 750.1780866559045 > 510.4803956111694.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -48.636811371129895 (T) = (0 -759.7560744194675) / Math.Sqrt((39.79007650873354 / (20)) + (116.85361866693587 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.5566444436132058 = (488.0729684525513 - 759.7560744194675) / 488.0729684525513 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Common.SplitWords(Options: IgnoreCase, Compiled)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 39.75759409987277 > 22.894813728030584.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -410.8153755648958 (T) = (0 -39857.92095836013) / Math.Sqrt((10693.32954216556 / (20)) + (5608.600750771533 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.8301724544164663 = (21778.23235300985 - 39857.92095836013) / 21778.23235300985 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Common.IP_IsNotMatch(Options: IgnoreCase, Compiled)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 761.44786205672 > 511.3735336568395.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -131.08879946781084 (T) = (0 -759.2021540632351) / Math.Sqrt((28.941570151254364 / (20)) + (11.480145625177942 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.5594804061637569 = (486.8301974571351 - 759.2021540632351) / 486.8301974571351 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Common.ReplaceWords(Options: Compiled)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 30.12709061692507 > 17.86990053159545.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -301.1483612156073 (T) = (0 -30022.13233205956) / Math.Sqrt((9066.295265202183 / (20)) + (5571.496613863267 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.7574685356437117 = (17082.600184966206 - 30022.13233205956) / 17082.600184966206 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWords(Options: Compiled)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 27.935903877520758 > 15.661154961621708.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -97.2747707239246 (T) = (0 -27917.600843324723) / Math.Sqrt((40080.295797277264 / (20)) + (62848.96395461995 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.8648916251707028 = (14970.092881815202 - 27917.600843324723) / 14970.092881815202 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchWord(Options: None)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.530762391684139 > 5.088140735880446.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -202.23585088070027 (T) = (0 -6549.797592526391) / Math.Sqrt((121.96437058723096 / (20)) + (260.22123269547257 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.3521781376491394 = (4843.886622744613 - 6549.797592526391) / 4843.886622744613 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWords(Options: IgnoreCase, Compiled)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 38.528310503685496 > 21.458525112080913.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -302.33464963516025 (T) = (0 -38416.74770887406) / Math.Sqrt((7669.0393259555885 / (20)) + (12656.184082225676 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.8823037074402957 = (20409.431037627914 - 38416.74770887406) / 20409.431037627914 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Common.SplitWords(Options: None)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 232.69811911764705 > 180.30689013534703.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -67.82813206706032 (T) = (0 -232851.30607876906) / Math.Sqrt((101759.63593747115 / (20)) + (3216436.1970930006 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.35507672438867566 = (171836.25243346774 - 232851.30607876906) / 171836.25243346774 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Common.SplitWords(Options: Compiled)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 29.561065049826105 > 17.226320223399508.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -157.9826240968781 (T) = (0 -29518.354804542716) / Math.Sqrt((14409.440932866553 / (20)) + (24578.602858431623 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.7967727772705989 = (16428.54075816019 - 29518.354804542716) / 16428.54075816019 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options: IgnoreCase, Compiled)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 741.1150646265045 > 580.2572279417716.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -100.44542916505759 (T) = (0 -746.3470870517443) / Math.Sqrt((20.92947664856624 / (20)) + (13.631456971932238 / (5))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (5) - 2, .025) and -0.353927861712436 = (551.2458293810212 - 746.3470870517443) / 551.2458293810212 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Common.ReplaceWords(Options: None)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 231.48741438065608 > 181.4097145963929.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -168.04887326801753 (T) = (0 -232424.88167283632) / Math.Sqrt((107784.08288962235 / (20)) + (482201.7094653125 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.34514682903127736 = (172787.7408299135 - 232424.88167283632) / 172787.7408299135 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Backtracking(Options: IgnoreCase, Compiled)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.0110508872662547 > 575.3918523816478.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -65.49240443083679 (T) = (0 -990.7638652362518) / Math.Sqrt((31.316863964640913 / (20)) + (219.33393125151287 / (5))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (5) - 2, .025) and -0.8036167239644483 = (549.3206245385098 - 990.7638652362518) / 549.3206245385098 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Common.IP_IsMatch(Options: IgnoreCase, Compiled)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 761.000345091397 > 515.5374668464614.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -69.94405712312853 (T) = (0 -766.1028537928282) / Math.Sqrt((25.881813833098917 / (20)) + (57.51945698595643 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.5660578911944107 = (489.1919118063587 - 766.1028537928282) / 489.1919118063587 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Common.OneNodeBacktracking(Options: Compiled)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 778.2788156288154 > 357.9045717079674.
IsChangePoint: Marked as a change because one of 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -61.17477503419956 (T) = (0 -780.852697338659) / Math.Sqrt((9.122866758079258 / (19)) + (205.035140150628 / (4))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (4) - 2, .025) and -1.2910739587695181 = (340.823871857038 - 780.852697338659) / 340.823871857038 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options: Compiled)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 755.8287323620373 > 573.8744367482457.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -54.5663417213032 (T) = (0 -749.2163602795785) / Math.Sqrt((26.44200871151724 / (20)) + (60.60151709804062 / (5))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (5) - 2, .025) and -0.36430774040234704 = (549.154958293081 - 749.2163602795785) / 549.154958293081 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Backtracking(Options: Compiled)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 800.7738887177908 > 661.6232957360315.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -44.677994244731885 (T) = (0 -792.8199730683116) / Math.Sqrt((69.1256957007973 / (20)) + (46.57791435065859 / (5))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (5) - 2, .025) and -0.25218175421462874 = (633.1508747830063 - 792.8199730683116) / 633.1508747830063 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Common.ReplaceWords(Options: IgnoreCase, Compiled)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 40.74478432720999 > 23.61974135299371.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -214.0677328952694 (T) = (0 -40563.936780098615) / Math.Sqrt((3113.0477755386273 / (20)) + (27930.688363213198 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.8046340357276547 = (22477.65251958281 - 40563.936780098615) / 22477.65251958281 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Common.IP_IsNotMatch(Options: Compiled)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 757.2619727926987 > 502.5721014028.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -79.06316819739882 (T) = (0 -757.2642812846641) / Math.Sqrt((70.08526001879336 / (20)) + (32.893545439376716 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.5565322790308207 = (486.50727741809305 - 757.2642812846641) / 486.50727741809305 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesWord(Options: IgnoreCase, Compiled)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.671891642011836 > 10.623149085099655.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -146.8946086684902 (T) = (0 -20740.28268084026) / Math.Sqrt((5374.037683260885 / (20)) + (19720.68428463209 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -1.0436373775244747 = (10148.709799956609 - 20740.28268084026) / 10148.709799956609 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsMatch(Options: Compiled)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 493.1404498868985 > 425.01726105386234.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -43.566428461641046 (T) = (0 -497.4694330250045) / Math.Sqrt((12.554183677770743 / (20)) + (18.899350266012732 / (5))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (5) - 2, .025) and -0.22527893799271242 = (406.0050471772357 - 497.4694330250045) / 406.0050471772357 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[performanceautofiler]

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	7a57f6dd800e77508a4ba405e623059935c82f43
Compare	887c043eb94be364188e2b23a87efa214ea57f1e
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Collections.ContainsTrue<Int32>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Array - Duration of single invocation	139.37 μs	426.96 μs	3.06	0.01	True
Queue - Duration of single invocation	133.53 μs	421.54 μs	3.16	0.01	True
ImmutableArray - Duration of single invocation	143.15 μs	436.56 μs	3.05	0.01	True
Span - Duration of single invocation	117.26 μs	413.86 μs	3.53	0.00	True
List - Duration of single invocation	134.95 μs	425.52 μs	3.15	0.01	True
Stack - Duration of single invocation	135.70 μs	434.16 μs	3.20	0.01	True
ICollection - Duration of single invocation	135.73 μs	426.97 μs	3.15	0.01	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

• Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
• CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
• AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
• Dotnet SDK installed for dotnet commands
• Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.ContainsTrue<Int32>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.ContainsTrue<Int32>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Collections.ContainsTrue<Int32>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Collections.ContainsTrue<Int32>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Details

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsTrue<Int32>.Array(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 426.9573567567567 > 146.33087115018986.
IsChangePoint: Marked as a change because one of 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -383.61410138160306 (T) = (0 -428541.8219494048) / Math.Sqrt((8997.850365136192 / (20)) + (2271976.8747673896 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -2.0760620412976327 = (139315.07758817016 - 428541.8219494048) / 139315.07758817016 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Collections.ContainsTrue<Int32>.Queue(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 421.5448289414414 > 141.3476764602675.
IsChangePoint: Marked as a change because one of 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -384.6472140474928 (T) = (0 -423443.82015988027) / Math.Sqrt((722137.5891788674 / (20)) + (2114105.723530897 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -2.1503452079501035 = (134411.879400166 - 423443.82015988027) / 134411.879400166 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Collections.ContainsTrue<Int32>.ImmutableArray(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 436.5604888513514 > 150.87111797448392.
IsChangePoint: Marked as a change because one of 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -775.7258968627362 (T) = (0 -435686.42916443234) / Math.Sqrt((116073.18964843462 / (20)) + (543781.0967509773 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -2.033414500865287 = (143629.04543383437 - 435686.42916443234) / 143629.04543383437 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Collections.ContainsTrue<Int32>.Span(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 413.8554461348685 > 121.99937287004775.
IsChangePoint: Marked as a change because one of 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -304.3834605892573 (T) = (0 -415671.7364788926) / Math.Sqrt((702882.9136752111 / (20)) + (3743314.3573790216 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -2.591472776143598 = (115738.51798070064 - 415671.7364788926) / 115738.51798070064 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Collections.ContainsTrue<Int32>.List(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 425.5160778153153 > 141.34247914126973.
IsChangePoint: Marked as a change because one of 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -1524.30789758396 (T) = (0 -425519.74887387385) / Math.Sqrt((23470.93995662514 / (20)) + (140966.40768358728 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -2.160430003566674 = (134639.82698356156 - 425519.74887387385) / 134639.82698356156 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Collections.ContainsTrue<Int32>.Stack(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 434.15578997747747 > 142.57136124296326.
IsChangePoint: Marked as a change because one of 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -89.80045397898175 (T) = (0 -424585.444563193) / Math.Sqrt((52974.32478442763 / (20)) + (41332550.204343796 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -2.124637323794232 = (135883.1123631401 - 424585.444563193) / 135883.1123631401 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Collections.ContainsTrue<Int32>.ICollection(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 426.967228490991 > 142.52276569773704.
IsChangePoint: Marked as a change because one of 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -729.4675396290472 (T) = (0 -425904.985670045) / Math.Sqrt((410355.31905112276 / (20)) + (550054.1890462372 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -2.1335288399082817 = (135918.642345258 - 425904.985670045) / 135918.642345258 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[performanceautofiler]

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	7a57f6dd800e77508a4ba405e623059935c82f43
Compare	887c043eb94be364188e2b23a87efa214ea57f1e
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Tests.Perf_String

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
CtorCharCount - Duration of single invocation	26.39 ns	51.04 ns	1.93	0.13	True
PadLeft - Duration of single invocation	1.58 μs	2.49 μs	1.58	0.01	True
ToUpperInvariant - Duration of single invocation	495.12 ns	645.45 ns	1.30	0.02	True
Format_OneArg - Duration of single invocation	2.00 μs	3.00 μs	1.50	0.03	False
PadLeft - Duration of single invocation	43.65 ns	73.21 ns	1.68	0.10	False
CtorCharCount - Duration of single invocation	89.85 ns	154.60 ns	1.72	0.01	True
IndexOfAny - Duration of single invocation	119.56 ns	211.47 ns	1.77	0.03	False
ToLowerInvariant - Duration of single invocation	503.92 ns	646.17 ns	1.28	0.01	False
Replace_String - Duration of single invocation	72.18 ns	160.63 ns	2.23	0.04	False
Split - Duration of single invocation	1.52 μs	1.79 μs	1.18	0.04	False
Replace_String - Duration of single invocation	226.69 ns	336.59 ns	1.48	0.03	False
Format_MultipleArgs - Duration of single invocation	891.83 ns	1.49 μs	1.67	0.02	False
Replace_String - Duration of single invocation	273.01 ns	445.00 ns	1.63	0.04	False
ToUpper - Duration of single invocation	515.38 ns	664.44 ns	1.29	0.02	True
Split - Duration of single invocation	1.16 μs	1.39 μs	1.19	0.05	False
Replace_Char - Duration of single invocation	30.69 ns	64.22 ns	2.09	0.05	True
ToLower - Duration of single invocation	522.65 ns	670.56 ns	1.28	0.01	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

• Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
• CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
• AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
• Dotnet SDK installed for dotnet commands
• Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_String*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_String* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_String*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_String* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Details

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_String.CtorCharCount(size: 10)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 51.03746494626241 > 28.857276840102603.
IsChangePoint: Marked as a change because one of 3/21/2023 2:53:21 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -54.49852341173199 (T) = (0 -50.64712358424011) / Math.Sqrt((0.5098966373533874 / (20)) + (0.6862511910480072 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.9144928980046337 = (26.454589430457908 - 50.64712358424011) / 26.454589430457908 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Tests.Perf_String.PadLeft(n: 2142)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.4945356729827037 > 1.6703151559197924.
IsChangePoint: Marked as a change because one of 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -64.63908225032965 (T) = (0 -2507.557357893601) / Math.Sqrt((87.717129832563 / (20)) + (784.0282610573718 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.5745794174150666 = (1592.525172220384 - 2507.557357893601) / 1592.525172220384 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Tests.Perf_String.ToUpperInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 645.4522775800713 > 521.3028581577743.
IsChangePoint: Marked as a change because one of 12/21/2022 10:31:08 AM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -10.1654232760909 (T) = (0 -671.5049494503291) / Math.Sqrt((3.4926669941011337 / (20)) + (1483.3712165214329 / (5))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (5) - 2, .025) and -0.3528538227448634 = (496.36179324081286 - 671.5049494503291) / 496.36179324081286 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Tests.Perf_String.Format_OneArg(s: "Testing {0}, {0:C}, {0:D5}, {0:E} - {0:F4}{0:G}{0:N} {0:X} !!", o: 8)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.9988483060372513 > 2.0996156822768537.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -132.04799907731962 (T) = (0 -2994.2287528476522) / Math.Sqrt((236.92023814991842 / (20)) + (177.24633652625303 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.49361741130932724 = (2004.6825446570463 - 2994.2287528476522) / 2004.6825446570463 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_String.PadLeft(n: 18)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 73.20802990030364 > 45.78562700630902.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -145.5441509397233 (T) = (0 -72.88822683995225) / Math.Sqrt((0.29007801133970706 / (20)) + (0.07977813149947308 / (3))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (3) - 2, .025) and -0.6801071927241494 = (43.383081243626044 - 72.88822683995225) / 43.383081243626044 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_String.CtorCharCount(size: 100)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 154.60164993671646 > 95.15571698640568.
IsChangePoint: Marked as a change because one of 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -59.7447757714685 (T) = (0 -153.89155684635278) / Math.Sqrt((1.985302294140433 / (20)) + (3.9737988694653237 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.6830052237914541 = (91.43854972693889 - 153.89155684635278) / 91.43854972693889 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Tests.Perf_String.IndexOfAny

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 211.46985281647414 > 125.64308041288668.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -238.68900887015712 (T) = (0 -211.5523330730404) / Math.Sqrt((2.106811911371863 / (20)) + (0.1665547648015813 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.7622894554292552 = (120.04403273326677 - 211.5523330730404) / 120.04403273326677 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_String.ToLowerInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 646.1658893294377 > 523.3214625625078.
IsChangePoint: Marked as a change because one of 12/21/2022 10:31:08 AM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -8.332420512482003 (T) = (0 -664.0915504040211) / Math.Sqrt((12.278838488904153 / (20)) + (1589.0895551288868 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.333827799003003 = (497.8840228854204 - 664.0915504040211) / 497.8840228854204 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence", oldValue: "bad", newValue: "nice")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 160.62935760016302 > 75.95588010082321.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -551.2843971247921 (T) = (0 -160.40453111568115) / Math.Sqrt((0.13203218274110928 / (20)) + (0.07591268331394212 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -1.220627716367558 = (72.23386879907385 - 160.40453111568115) / 72.23386879907385 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_String.Split(s: "A B C D E F G H I J K L M N O P Q R S T U V W X Y Z", arr: [' '], options: RemoveEmptyEntries)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.7927435793057156 > 1.5894301319080353.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -28.619839689129943 (T) = (0 -1806.198543922144) / Math.Sqrt((189.67524023975216 / (20)) + (390.2282076809193 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.19608180398362463 = (1510.0961639132772 - 1806.198543922144) / 1510.0961639132772 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence", oldValue: "nice", newValue: "bad")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 336.58909233459656 > 238.7993567326821.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -36.17278256497747 (T) = (0 -330.883201279019) / Math.Sqrt((20.78088693336406 / (20)) + (28.13073128638034 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.4505201230569503 = (228.11348565208968 - 330.883201279019) / 228.11348565208968 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_String.Format_MultipleArgs

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4906783451600916 > 931.7996001720571.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -62.415093578796125 (T) = (0 -1494.2966422367365) / Math.Sqrt((38.721567540010994 / (20)) + (366.96068814243637 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.6786024086343593 = (890.2028464575085 - 1494.2966422367365) / 890.2028464575085 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence. This is another very nice sentence.", oldValue: "a", newValue: "")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 445.0031207302805 > 286.3648279707919.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -149.90418948106205 (T) = (0 -443.17786184212446) / Math.Sqrt((0.8522317582096567 / (20)) + (4.986785953840935 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.6235696948706662 = (272.96509859863335 - 443.17786184212446) / 272.96509859863335 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_String.ToUpper(s: "This is a much longer piece of text that might benefit more from vectorization.")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 664.4395120200933 > 542.6496054551927.
IsChangePoint: Marked as a change because one of 12/21/2022 10:31:08 AM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -9.116232511483737 (T) = (0 -695.6467966256208) / Math.Sqrt((16.455607286873157 / (20)) + (1915.637505760481 / (5))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (5) - 2, .025) and -0.34549973175416365 = (517.0174175499 - 695.6467966256208) / 517.0174175499 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Tests.Perf_String.Split(s: "A B C D E F G H I J K L M N O P Q R S T U V W X Y Z", arr: [' '], options: None)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.3895804811463714 > 1.2527339114836595.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -64.07678249678291 (T) = (0 -1391.310798923104) / Math.Sqrt((173.10629085956094 / (20)) + (5.572275409445161 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.1709471813679689 = (1188.1926196685433 - 1391.310798923104) / 1188.1926196685433 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_String.Replace_Char(text: "This is a very nice sentence", oldChar: 'z', newChar: 'y')

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 64.22108062264009 > 32.22459561218911.
IsChangePoint: Marked as a change because one of 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -418.92444900543575 (T) = (0 -64.4226923391605) / Math.Sqrt((0.0012905042977808517 / (20)) + (0.025689582410493326 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -1.099695522636291 = (30.681921090289336 - 64.4226923391605) / 30.681921090289336 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Tests.Perf_String.ToLower(s: "This is a much longer piece of text that might benefit more from vectorization.")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 670.5625982182984 > 549.9918173080408.
IsChangePoint: Marked as a change because one of 12/21/2022 10:31:08 AM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -137.35629093345747 (T) = (0 -669.8141197008794) / Math.Sqrt((19.174830038929755 / (20)) + (0.43098736675930527 / (3))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (3) - 2, .025) and -0.2743889643851093 = (525.5962962799695 - 669.8141197008794) / 525.5962962799695 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[performanceautofiler]

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	7a57f6dd800e77508a4ba405e623059935c82f43
Compare	887c043eb94be364188e2b23a87efa214ea57f1e
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Globalization.Tests.StringSearch

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
IsSuffix_SecondHalf - Duration of single invocation	511.26 ns	1.72 μs	3.36	0.01	False
IndexOf_Word_NotFound - Duration of single invocation	416.92 ns	643.41 ns	1.54	0.02	False
IsSuffix_DifferentLastChar - Duration of single invocation	998.43 ns	3.40 μs	3.41	0.01	False
LastIndexOf_Word_NotFound - Duration of single invocation	294.11 ns	649.23 ns	2.21	0.01	True
IndexOf_Word_NotFound - Duration of single invocation	292.02 ns	636.68 ns	2.18	0.02	True
IsPrefix_FirstHalf - Duration of single invocation	510.27 ns	1.72 μs	3.37	0.01	False
IsPrefix_DifferentFirstChar - Duration of single invocation	37.96 ns	149.68 ns	3.94	0.07	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

• Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
• CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
• AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
• Dotnet SDK installed for dotnet commands
• Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Globalization.Tests.StringSearch*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Globalization.Tests.StringSearch* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Globalization.Tests.StringSearch*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Globalization.Tests.StringSearch* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Details

Payloads

Baseline
Compare

Histogram

System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, OrdinalIgnoreCase, False))

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.718622237676644 > 537.2384344328472.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -146.21151149340403 (T) = (0 -1740.8098196986239) / Math.Sqrt((0.9784733482528948 / (20)) + (282.6142247114641 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -2.404049940010073 = (511.393736983034 - 1740.8098196986239) / 511.393736983034 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, OrdinalIgnoreCase, False))

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 643.4117315036299 > 437.3712564061474.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -274.1419732532332 (T) = (0 -643.4717430821356) / Math.Sqrt((13.430725091209638 / (20)) + (0.014527910696356049 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.5386218162838027 = (418.21306332201755 - 643.4717430821356) / 418.21306332201755 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (en-US, OrdinalIgnoreCase, False))

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.4016420149539877 > 1.0500891555179235.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -188.58797232055613 (T) = (0 -3427.3703460229804) / Math.Sqrt((1.372518323420927 / (20)) + (496.7774072869987 / (3))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (3) - 2, .025) and -2.427144564554151 = (1000.0658803457445 - 3427.3703460229804) / 1000.0658803457445 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (en-US, Ordinal, False))

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 649.2286342989088 > 308.57026827891787.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -196.5604198202181 (T) = (0 -654.3596929590547) / Math.Sqrt((0.9090885410147147 / (20)) + (16.61570531612036 / (5))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (5) - 2, .025) and -1.2287612741848193 = (293.5979283821638 - 654.3596929590547) / 293.5979283821638 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, Ordinal, False))

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 636.6820894589981 > 305.9350846739994.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -326.2988180798156 (T) = (0 -639.7358893403639) / Math.Sqrt((0.5656760684096057 / (20)) + (4.454131339453421 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -1.1979020184207938 = (291.0666098755476 - 639.7358893403639) / 291.0666098755476 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, OrdinalIgnoreCase, False))

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.7200651947943373 > 535.3830255233592.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -127.99528822757563 (T) = (0 -1748.9538569130677) / Math.Sqrt((2.350677782711305 / (20)) + (374.3168195232247 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -2.4293421621485716 = (509.9968956778879 - 1748.9538569130677) / 509.9968956778879 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en-US, OrdinalIgnoreCase, False))

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 149.67749857395899 > 39.70219039253499.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -89.85364900589968 (T) = (0 -148.40101119783782) / Math.Sqrt((1.0196027606112175 / (20)) + (5.8980431332600345 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -2.9656001376685497 = (37.42208141163863 - 148.40101119783782) / 37.42208141163863 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	7a57f6dd800e77508a4ba405e623059935c82f43
Compare	887c043eb94be364188e2b23a87efa214ea57f1e
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.IO.Tests.BinaryWriterExtendedTests

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
WriteAsciiString - Duration of single invocation	126.69 ns	188.51 ns	1.49	0.02	False
WriteAsciiCharArray - Duration of single invocation	183.61 ns	246.55 ns	1.34	0.02	False
WriteAsciiCharArray - Duration of single invocation	4.49 ms	5.51 ms	1.23	0.01	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

• Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
• CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
• AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
• Dotnet SDK installed for dotnet commands
• Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.BinaryWriterExtendedTests*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.IO.Tests.BinaryWriterExtendedTests* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.IO.Tests.BinaryWriterExtendedTests*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.IO.Tests.BinaryWriterExtendedTests* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Details

Payloads

Baseline
Compare

Histogram

System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiString(StringLengthInChars: 32)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 188.5136208936323 > 132.6139239364662.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -3.851895185400494 (T) = (0 -235.77343474167128) / Math.Sqrt((1.4180272740778979 / (19)) + (3987.4023081472524 / (5))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (5) - 2, .025) and -0.8566007816945429 = (126.99199368346592 - 235.77343474167128) / 126.99199368346592 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiCharArray(StringLengthInChars: 32)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 246.54812297486816 > 191.20304228026225.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -3.7821285350471596 (T) = (0 -293.60628058513714) / Math.Sqrt((1.2273885474899595 / (20)) + (4315.9405800937775 / (5))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (5) - 2, .025) and -0.6089500454452816 = (182.48315503411465 - 293.60628058513714) / 182.48315503411465 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiCharArray(StringLengthInChars: 2000000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.510790876388889 > 4.7119283935096155.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -2.4325022046056803 (T) = (0 -7393923.10777778) / Math.Sqrt((124828631.41175358 / (20)) + (7122553893843.43 / (5))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (5) - 2, .025) and -0.6465130881157805 = (4490655.532073031 - 7393923.10777778) / 4490655.532073031 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	7a57f6dd800e77508a4ba405e623059935c82f43
Compare	887c043eb94be364188e2b23a87efa214ea57f1e
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Tests.Perf_Guid

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
EqualsSame - Duration of single invocation	2.25 ns	11.85 ns	5.28	0.36	False
NotEqualsOperator - Duration of single invocation	3.62 ns	14.11 ns	3.89	0.22	False
EqualsOperator - Duration of single invocation	3.53 ns	14.57 ns	4.13	0.21	False
EqualsNotSame - Duration of single invocation	0.79 ns	12.07 ns	15.27	0.66	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

• Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
• CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
• AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
• Dotnet SDK installed for dotnet commands
• Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Guid*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Guid* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Guid*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Guid* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Details

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Guid.EqualsSame

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.854975206270687 > 2.488085405366373.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 3:33:39 AM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -107.9611990277699 (T) = (0 -11.997258084734971) / Math.Sqrt((0.02464205926655041 / (20)) + (0.02769458827850055 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -4.338355814207272 = (2.2473695089424313 - 11.997258084734971) / 2.2473695089424313 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_Guid.NotEqualsOperator

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.10509988122932 > 3.870186718962021.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -76.50568407586567 (T) = (0 -13.797750976803083) / Math.Sqrt((0.01967894857738079 / (20)) + (0.06494603050817126 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -2.671411156628791 = (3.758160115597793 - 13.797750976803083) / 3.758160115597793 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_Guid.EqualsOperator

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.57257624518515 > 3.791397661162461.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -31.61238069155947 (T) = (0 -13.844084760129434) / Math.Sqrt((0.04265080762621711 / (20)) + (0.4048783867659959 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -2.760751617079849 = (3.681201570785762 - 13.844084760129434) / 3.681201570785762 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_Guid.EqualsNotSame

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.068243251037416 > 0.7390947732403752.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 3:33:39 AM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -82.1121189090944 (T) = (0 -12.102101276826858) / Math.Sqrt((0.30118391754002694 / (20)) + (0.010914128815051391 / (3))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (3) - 2, .025) and -12.84257026215852 = (0.8742669206390385 - 12.102101276826858) / 0.8742669206390385 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[performanceautofiler]

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	7a57f6dd800e77508a4ba405e623059935c82f43
Compare	887c043eb94be364188e2b23a87efa214ea57f1e
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Memory.Span<Char>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Fill - Duration of single invocation	6.26 ns	8.95 ns	1.43	0.02	False
IndexOfAnyTwoValues - Duration of single invocation	32.20 ns	116.99 ns	3.63	0.03	True
SequenceCompareToDifferent - Duration of single invocation	27.88 ns	48.77 ns	1.75	0.04	False
IndexOfValue - Duration of single invocation	22.30 ns	66.20 ns	2.97	0.06	True
Reverse - Duration of single invocation	35.89 ns	336.43 ns	9.37	0.02	True
IndexOfAnyThreeValues - Duration of single invocation	43.30 ns	152.09 ns	3.51	0.07	True
Reverse - Duration of single invocation	441.71 ns	5.03 μs	11.38	0.01	False
Fill - Duration of single invocation	21.56 ns	58.28 ns	2.70	0.02	True
LastIndexOfValue - Duration of single invocation	257.23 ns	431.39 ns	1.68	0.01	True
Fill - Duration of single invocation	276.98 ns	517.94 ns	1.87	0.00	True
LastIndexOfAnyValues - Duration of single invocation	404.07 ns	497.04 ns	1.23	0.01	False
IndexOfValue - Duration of single invocation	257.85 ns	442.26 ns	1.72	0.01	True
IndexOfAnyFourValues - Duration of single invocation	57.70 ns	177.03 ns	3.07	0.01	True
LastIndexOfAnyValues - Duration of single invocation	33.46 ns	107.94 ns	3.23	0.05	True
IndexOfAnyTwoValues - Duration of single invocation	404.89 ns	525.20 ns	1.30	0.02	False
SequenceCompareToDifferent - Duration of single invocation	28.00 ns	48.31 ns	1.73	0.05	False
LastIndexOfValue - Duration of single invocation	24.09 ns	68.20 ns	2.83	0.08	True
IndexOfAnyFiveValues - Duration of single invocation	67.00 ns	198.98 ns	2.97	0.02	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

• Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
• CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
• AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
• Dotnet SDK installed for dotnet commands
• Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.Span<Char>*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Memory.Span<Char>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.Span<Char>*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Memory.Span<Char>* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Details

Payloads

Baseline
Compare

Histogram

System.Memory.Span<Char>.Fill(Size: 4)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.950295487581034 > 6.643944527895347.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 3/21/2023 2:53:21 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -34.47181981513003 (T) = (0 -8.914639865643883) / Math.Sqrt((0.007349549872758054 / (20)) + (0.026807407792185715 / (5))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (5) - 2, .025) and -0.41379428556312603 = (6.305471705944198 - 8.914639865643883) / 6.305471705944198 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Memory.Span<Char>.IndexOfAnyTwoValues(Size: 33)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 116.98684398940604 > 34.301364561946514.
IsChangePoint: Marked as a change because one of 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -321.4369755774487 (T) = (0 -116.60212619153371) / Math.Sqrt((0.28695179981194674 / (19)) + (0.2642171590192417 / (5))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (5) - 2, .025) and -2.5533385809578237 = (32.81480881568649 - 116.60212619153371) / 32.81480881568649 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Memory.Span<Char>.SequenceCompareToDifferent(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 48.771238402010425 > 29.284770500555965.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -137.20654626871848 (T) = (0 -48.45875819711011) / Math.Sqrt((0.17337512992349305 / (20)) + (0.05384427115452356 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.7277216917779055 = (28.0477801648967 - 48.45875819711011) / 28.0477801648967 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Memory.Span<Char>.IndexOfValue(Size: 33)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 66.19577308406495 > 23.456884468801714.
IsChangePoint: Marked as a change because one of 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -50.13600665650588 (T) = (0 -64.55614116294655) / Math.Sqrt((0.36784765123592233 / (20)) + (3.437751036064798 / (5))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (5) - 2, .025) and -1.8779013182780422 = (22.431672953113257 - 64.55614116294655) / 22.431672953113257 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Memory.Span<Char>.Reverse(Size: 33)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 336.4299810874198 > 37.90837435355754.
IsChangePoint: Marked as a change because one of 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -182.7608592287904 (T) = (0 -335.9677046129287) / Math.Sqrt((0.09758907351929158 / (19)) + (10.742125875115306 / (4))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (4) - 2, .025) and -8.285901649891757 = (36.180407383147866 - 335.9677046129287) / 36.180407383147866 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Memory.Span<Char>.IndexOfAnyThreeValues(Size: 33)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 152.09138371628737 > 45.72947003839956.
IsChangePoint: Marked as a change because one of 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -162.7995437571347 (T) = (0 -150.9483719840178) / Math.Sqrt((5.06885158293491 / (20)) + (0.8868186231002737 / (5))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (5) - 2, .025) and -2.423367860951146 = (44.09352956362639 - 150.9483719840178) / 44.09352956362639 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Memory.Span<Char>.Reverse(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.025292742531188 > 463.96951519556393.
IsChangePoint: Marked as a change because one of 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -195.35635645952723 (T) = (0 -5072.5883585595375) / Math.Sqrt((0.09920266625983949 / (20)) + (1685.5252253969315 / (3))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (3) - 2, .025) and -10.47652428557231 = (441.9969175629708 - 5072.5883585595375) / 441.9969175629708 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Memory.Span<Char>.Fill(Size: 33)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 58.281195729600014 > 22.45173388389547.
IsChangePoint: Marked as a change because one of 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -187.18126237874907 (T) = (0 -57.72527962617653) / Math.Sqrt((0.008354805170918135 / (20)) + (0.18577073778305986 / (5))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (5) - 2, .025) and -1.6920265543100261 = (21.44305728847897 - 57.72527962617653) / 21.44305728847897 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Memory.Span<Char>.LastIndexOfValue(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 431.3897889978346 > 270.11388827011467.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -52.71234911352573 (T) = (0 -436.97985999353097) / Math.Sqrt((7.674445071221518 / (20)) + (44.84438036771714 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.6970870776681324 = (257.4881782695319 - 436.97985999353097) / 257.4881782695319 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Memory.Span<Char>.Fill(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 517.944582268478 > 290.37425760387254.
IsChangePoint: Marked as a change because one of 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -135.03986733412725 (T) = (0 -513.8804034469283) / Math.Sqrt((0.2337394136513964 / (20)) + (15.359308981176909 / (5))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (5) - 2, .025) and -0.8568403935078159 = (276.7499055081092 - 513.8804034469283) / 276.7499055081092 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Memory.Span<Char>.LastIndexOfAnyValues(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 497.03907597391975 > 424.1285711294525.
IsChangePoint: Marked as a change because one of 3/19/2023 2:30:51 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -39.875393760172095 (T) = (0 -498.07103052690275) / Math.Sqrt((55.4957252374912 / (20)) + (9.390678671859492 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.22129611716040923 = (407.8216769287283 - 498.07103052690275) / 407.8216769287283 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Memory.Span<Char>.IndexOfValue(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 442.2595725898764 > 270.5214316167896.
IsChangePoint: Marked as a change because one of 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -336.23354472396244 (T) = (0 -443.0020629349684) / Math.Sqrt((0.6427407786257856 / (20)) + (1.0939646003121755 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.7229396946916167 = (257.1198889316088 - 443.0020629349684) / 257.1198889316088 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Memory.Span<Char>.IndexOfAnyFourValues(Size: 33)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 177.02796576813185 > 60.64281491311468.
IsChangePoint: Marked as a change because one of 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -148.87845479618744 (T) = (0 -176.9970830169096) / Math.Sqrt((0.07605266675610946 / (19)) + (2.551105098885331 / (4))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (4) - 2, .025) and -2.0660050140438275 = (57.72889548652888 - 176.9970830169096) / 57.72889548652888 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Memory.Span<Char>.LastIndexOfAnyValues(Size: 33)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 107.93925955821177 > 34.495303692015455.
IsChangePoint: Marked as a change because one of 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -379.77129305243767 (T) = (0 -107.65790940407899) / Math.Sqrt((0.09636277693224415 / (20)) + (0.13492628716355223 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -2.2531624219161146 = (33.09330904562349 - 107.65790940407899) / 33.09330904562349 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Memory.Span<Char>.IndexOfAnyTwoValues(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 525.197025719774 > 425.66643831376115.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -99.71426300288435 (T) = (0 -528.2753767051058) / Math.Sqrt((15.342841093513059 / (20)) + (3.645561384837533 / (5))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (5) - 2, .025) and -0.3001991906407425 = (406.3034191282414 - 528.2753767051058) / 406.3034191282414 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Memory.Span<Char>.SequenceCompareToDifferent(Size: 33)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 48.31219192134226 > 29.434941540780475.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -91.11923871404895 (T) = (0 -48.197690124658095) / Math.Sqrt((0.9091468933467792 / (20)) + (0.009726123801168976 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.7056548509405103 = (28.25758687232739 - 48.197690124658095) / 28.25758687232739 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Memory.Span<Char>.LastIndexOfValue(Size: 33)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 68.19868138022895 > 24.5779624004318.
IsChangePoint: Marked as a change because one of 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -401.62386159511675 (T) = (0 -68.29208657867136) / Math.Sqrt((0.12104829545533996 / (20)) + (0.025615856072951883 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -1.9100595153683606 = (23.467591029672402 - 68.29208657867136) / 23.467591029672402 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Memory.Span<Char>.IndexOfAnyFiveValues(Size: 33)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 198.97688461313163 > 70.42484419906711.
IsChangePoint: Marked as a change because one of 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -282.8166885480506 (T) = (0 -199.77790924244266) / Math.Sqrt((0.4293717040969332 / (20)) + (0.7959644217272818 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -1.982355996269405 = (66.98660706245082 - 199.77790924244266) / 66.98660706245082 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[performanceautofiler]

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	7a57f6dd800e77508a4ba405e623059935c82f43
Compare	887c043eb94be364188e2b23a87efa214ea57f1e
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Tests.Perf_Enum

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
StringFormat - Duration of single invocation	1.14 μs	1.55 μs	1.35	0.02	False
StringFormat - Duration of single invocation	934.64 ns	1.32 μs	1.41	0.01	False
Parse_Flags - Duration of single invocation	1.04 μs	1.42 μs	1.36	0.01	False
IsDefined_Generic_Flags - Duration of single invocation	113.02 ns	144.66 ns	1.28	0.41	False
TryParseGeneric_Flags - Duration of single invocation	1.08 μs	1.42 μs	1.31	0.02	False
GetName_Generic_Flags - Duration of single invocation	117.39 ns	156.96 ns	1.34	0.48	False
GetName_NonGeneric_Flags - Duration of single invocation	181.48 ns	208.25 ns	1.15	0.16	False
StringFormat - Duration of single invocation	1.26 μs	1.61 μs	1.29	0.01	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

• Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
• CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
• AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
• Dotnet SDK installed for dotnet commands
• Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Enum*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Enum* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_Enum*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_Enum* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Details

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Enum.StringFormat(value: 32)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.5454549275434077 > 1.2060366325799308.
IsChangePoint: Marked as a change because one of 1/15/2023 4:47:52 PM, 2/12/2023 11:53:55 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -112.65855057646539 (T) = (0 -1546.4600756140615) / Math.Sqrt((94.70016883432045 / (20)) + (40.03449193075392 / (5))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (5) - 2, .025) and -0.3514271375869626 = (1144.3162806211953 - 1546.4600756140615) / 1144.3162806211953 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_Enum.StringFormat(value: Red)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.3150569992767804 > 980.9750779407526.
IsChangePoint: Marked as a change because one of 2/12/2023 11:53:55 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -55.79473428787901 (T) = (0 -1339.660841294097) / Math.Sqrt((77.73379997018769 / (19)) + (244.0911221316902 / (5))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (5) - 2, .025) and -0.4346088861753321 = (933.8160764259821 - 1339.660841294097) / 933.8160764259821 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_Enum.Parse_Flags(text: "Red, Orange, Yellow, Green, Blue")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4196337297139783 > 1.1058806183432464.
IsChangePoint: Marked as a change because one of 4/1/2023 8:54:51 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -27.125778146109607 (T) = (0 -1384.6964944586048) / Math.Sqrt((38.5985140826377 / (20)) + (590.2975647781169 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.31497108112393724 = (1053.0242940970775 - 1384.6964944586048) / 1053.0242940970775 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_Enum.IsDefined_Generic_Flags

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 144.66172163721427 > 120.68949047726272.
IsChangePoint: Marked as a change because one of 3/25/2023 2:21:39 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -21.86740559549453 (T) = (0 -147.70807979542332) / Math.Sqrt((19.963770164933173 / (19)) + (4.351979804188723 / (4))) is less than -2.079613844716807 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (4) - 2, .025) and -0.27633396538659466 = (115.72839382260217 - 147.70807979542332) / 115.72839382260217 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_Enum.TryParseGeneric_Flags(text: "Red, Orange, Yellow, Green, Blue")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4163496415197734 > 1.140091710821603.
IsChangePoint: Marked as a change because one of 4/1/2023 8:54:51 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -52.733356634500524 (T) = (0 -1411.8045416805894) / Math.Sqrt((191.48826174959208 / (20)) + (110.50922449849652 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.2950352758067437 = (1090.1668611313341 - 1411.8045416805894) / 1090.1668611313341 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_Enum.GetName_Generic_Flags

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 156.96075778275232 > 121.88542592708049.
IsChangePoint: Marked as a change because one of 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -15.904600251829505 (T) = (0 -153.8846532427324) / Math.Sqrt((18.793469456006274 / (20)) + (16.2217835881863 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.3003796867459674 = (118.33824752200562 - 153.8846532427324) / 118.33824752200562 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_Enum.GetName_NonGeneric_Flags

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 208.2503156130136 > 188.85565430429563.
IsChangePoint: Marked as a change because one of 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -9.39927747558601 (T) = (0 -207.8366940258748) / Math.Sqrt((22.01076005359073 / (20)) + (37.183640040989076 / (5))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (5) - 2, .025) and -0.1522585102466822 = (180.3733208977384 - 207.8366940258748) / 180.3733208977384 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_Enum.StringFormat(value: Red, Green)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.614568790176257 > 1.3165310163309534.
IsChangePoint: Marked as a change because one of 1/15/2023 4:47:52 PM, 2/12/2023 11:53:55 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -32.70494383194578 (T) = (0 -1632.1389755314717) / Math.Sqrt((95.10315669945004 / (20)) + (650.1509962547368 / (5))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (5) - 2, .025) and -0.30316315759620716 = (1252.4440750320841 - 1632.1389755314717) / 1252.4440750320841 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	7a57f6dd800e77508a4ba405e623059935c82f43
Compare	887c043eb94be364188e2b23a87efa214ea57f1e
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Memory.ReadOnlySpan

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
IndexOfString - Duration of single invocation	32.25 ns	92.01 ns	2.85	0.05	True
IndexOfString - Duration of single invocation	63.15 ns	123.33 ns	1.95	0.01	True
IndexOfString - Duration of single invocation	192.70 ns	318.47 ns	1.65	0.01	True
IndexOfString - Duration of single invocation	493.94 ns	822.66 ns	1.67	0.01	True
IndexOfString - Duration of single invocation	137.40 ns	206.16 ns	1.50	0.01	False
IndexOfString - Duration of single invocation	493.75 ns	820.21 ns	1.66	0.01	True
IndexOfString - Duration of single invocation	61.44 ns	97.36 ns	1.58	0.01	True

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

• Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
• CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
• AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
• Dotnet SDK installed for dotnet commands
• Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.ReadOnlySpan*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Memory.ReadOnlySpan* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Memory.ReadOnlySpan*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Memory.ReadOnlySpan* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Details

Payloads

Baseline
Compare

Histogram

System.Memory.ReadOnlySpan.IndexOfString(input: "Hello WorldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylongHello WorldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylongHello Worldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylong!xyz", value: "w", comparisonType: OrdinalIgnoreCase)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 92.00593737426348 > 33.98143667051142.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -452.70921195441883 (T) = (0 -91.68483247838017) / Math.Sqrt((0.021182399925792322 / (20)) + (0.06438339332538356 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -1.8306219611985837 = (32.39034874143265 - 91.68483247838017) / 32.39034874143265 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Memory.ReadOnlySpan.IndexOfString(input: "だだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだすだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだ", value: "す", comparisonType: Ordinal)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 123.33039888291916 > 66.33731077239888.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -130.91737871469198 (T) = (0 -122.61362626593996) / Math.Sqrt((0.007096959229179692 / (20)) + (0.8222399463150287 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.9399003164639291 = (63.206147875392574 - 122.61362626593996) / 63.206147875392574 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Memory.ReadOnlySpan.IndexOfString(input: "Hello WorldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylongHello WorldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylongHello Worldbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbareallyreallylong!xyz", value: "~", comparisonType: Ordinal)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 318.46984071833447 > 202.5297191840552.
IsChangePoint: Marked as a change because one of 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -345.1492167704594 (T) = (0 -319.0542497393829) / Math.Sqrt((0.023908648113305665 / (20)) + (0.5298019064156603 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.6541941936699803 = (192.87593376901657 - 319.0542497393829) / 192.87593376901657 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAXAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", value: "X", comparisonType: Ordinal)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 822.6635269697848 > 519.3522476073997.
IsChangePoint: Marked as a change because one of 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -438.3874941587993 (T) = (0 -822.3535919923667) / Math.Sqrt((0.41066683730571496 / (20)) + (2.154811146891746 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.6629388044684739 = (494.51825273583415 - 822.3535919923667) / 494.51825273583415 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAXAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", value: "x", comparisonType: OrdinalIgnoreCase)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 206.16457896911677 > 144.35845897868256.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -37.680777842114246 (T) = (0 -207.2870685172805) / Math.Sqrt((1.725478089804141 / (20)) + (13.134352095634876 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.5008211211127984 = (138.11577249365033 - 207.2870685172805) / 138.11577249365033 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Memory.ReadOnlySpan.IndexOfString(input: "だだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだxだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだだ", value: "x", comparisonType: Ordinal)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 820.212367407703 > 519.7480518418936.
IsChangePoint: Marked as a change because one of 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -533.3219112789692 (T) = (0 -821.5968248123885) / Math.Sqrt((0.4586465049126128 / (20)) + (1.7635102067794808 / (5))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (5) - 2, .025) and -0.66070053786848 = (494.7290652816391 - 821.5968248123885) / 494.7290652816391 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

System.Memory.ReadOnlySpan.IndexOfString(input: "Hello Worldbbbbbbbbbbbbbbcbbbbbbbbbbbbbbbbbbba!", value: "y", comparisonType: Ordinal)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 97.36452506011233 > 64.36720887531726.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/12/2023 6:21:10 AM, 4/18/2023 2:18:01 AM falls between 4/9/2023 1:58:52 PM and 4/18/2023 2:18:01 AM.
IsRegressionStdDev: Marked as regression because -77.2867309127093 (T) = (0 -98.07484833363725) / Math.Sqrt((0.13615651559617312 / (20)) + (0.8756736500679838 / (4))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (4) - 2, .025) and -0.5984709241236486 = (61.35541588747143 - 98.07484833363725) / 61.35541588747143 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository