[legacy x86 ryujit][Windows] SeekUnroll and Serialization perf tests are broken

[jorive]

Steps to repro:

set "__TestIntermediateDir=int"
.\build.cmd release x86
.\tests\runtest.cmd release x86 GenerateLayoutOnly

set /a "XUNIT_PERFORMANCE_MAX_ITERATION=2"
set /a "XUNIT_PERFORMANCE_MAX_ITERATION_INNER_SPECIFIED=2"

.\tests\scripts\run-xunit-perf.cmd -arch x86 -configuration Release -runtype local -testEnv "C:\github\dotnet\coreclr\tests\legacyjit_x86_testenv.cmd" -optLevel full_opt -jitName legacy_backend -testBinLoc bin\tests\Windows_NT.x86.Release\JIT\Performance\CodeQuality

Result dotnet/coreclr#1 (optionally, after running the repro steps above, you can run the command below):

username@computername "C:\github\dotnet\coreclr\bin\sandbox"
[Thu 10/12/2017][13:09:12] $ corerun.exe PerfHarness.dll "C:\github\dotnet\coreclr\bin\sandbox\Deserialize.exe" --perf:outputdir "C:\github\dotnet\coreclr\bin\sandbox\Logs" --perf:runid "Perf-Off" --perf:collect stopwatch
[10/12/2017 1:09:13 PM][INF] Running 4 [Benchmark]s
[10/12/2017 1:09:13 PM][INF]   Serialization.JsonBenchmarks.DeserializeDataContract
[10/12/2017 1:09:13 PM][ERR] Serialization.JsonBenchmarks.DeserializeDataContract: Error in line 1 position 142. Expecting element 'JsonBenchmarks.TestObject' from namespace 'http:///schemas.datacontract.org/2004/07/Serialization'.. Encountered 'Element'  with name 'JsonBenchmarks.TestObject', namespace 'http://schemas.datacontract.org/2004/07/Serialization'.
[10/12/2017 1:09:13 PM][ERR]    at System.Runtime.Serialization.DataContractSerializer.InternalReadObject(XmlReaderDelegator xmlReader, Boolean verifyObjectName, DataContractResolver dataContractResolver) in E:\A\_work\1317\s\corefx\src\System.Private.DataContractSerialization\src\System\Runtime\Serialization\DataContractSerializer.cs:line 446
   at System.Runtime.Serialization.XmlObjectSerializer.ReadObjectHandleExceptions(XmlReaderDelegator reader, Boolean verifyObjectName, DataContractResolver dataContractResolver) in E:\A\_work\1317\s\corefx\src\System.Private.DataContractSerialization\src\System\Runtime\Serialization\XmlObjectSerializer.cs:line 280
   at System.Runtime.Serialization.XmlObjectSerializer.ReadObject(XmlDictionaryReader reader) in E:\A\_work\1317\s\corefx\src\System.Private.DataContractSerialization\src\System\Runtime\Serialization\XmlObjectSerializer.cs:line 235
   at System.Runtime.Serialization.XmlObjectSerializer.ReadObject(Stream stream) in E:\A\_work\1317\s\corefx\src\System.Private.DataContractSerialization\src\System\Runtime\Serialization\XmlObjectSerializer.cs:line 223
   at Serialization.JsonBenchmarks.DeserializeDataContractBenchInner(DataContractSerializer ds, MemoryStream ms)
   at Serialization.JsonBenchmarks.DeserializeDataContractBench()
   at Serialization.JsonBenchmarks.DeserializeDataContract()
[10/12/2017 1:09:13 PM][INF]   Serialization.JsonBenchmarks.DeserializeJsonNetBinary
[10/12/2017 1:09:20 PM][INF]   Serialization.JsonBenchmarks.DeserializeDataContractJson
[10/12/2017 1:10:27 PM][INF]   Serialization.JsonBenchmarks.DeserializeJsonNet
[10/12/2017 1:10:35 PM][INF] Finished 4 tests in 82.187s (1 failed, 0 skipped)

Result dotnet/coreclr#2 (optionally, after running the repro steps above, you can run the command below):

username@computername "C:\github\dotnet\coreclr\bin\sandbox"
[Thu 10/12/2017][13:19:17] $ corerun.exe PerfHarness.dll "C:\github\dotnet\coreclr\bin\sandbox\SeekUnroll.exe" --perf:outputdir "C:\github\dotnet\coreclr\bin\sandbox\Logs" --perf:runid "Perf-Off" --perf:collect stopwatch
[10/12/2017 1:19:18 PM][INF] Running 5 [Benchmark]s
[10/12/2017 1:19:18 PM][INF]   SeekUnroll.Test(boxedIndex: 3)
[10/12/2017 1:19:45 PM][INF]   SeekUnroll.Test(boxedIndex: 1)
[10/12/2017 1:20:12 PM][INF]   SeekUnroll.Test(boxedIndex: 27)
[10/12/2017 1:20:54 PM][ERR] SeekUnroll.Test(boxedIndex: 27): Assert.Equal() Failure
Expected: 27
Actual:   159
[10/12/2017 1:20:54 PM][ERR]    at Xunit.Assert.Equal[T](T expected, T actual, IEqualityComparer`1 comparer) in C:\BuildAgent\work\cb37e9acf085d108\src\xunit.assert\Asserts\EqualityAsserts.cs:line 35
   at SeekUnroll.Test(Int32 index, Boolean isXunitBenchmark)
   at SeekUnroll.Test(Object boxedIndex)
[10/12/2017 1:20:54 PM][INF]   SeekUnroll.Test(boxedIndex: 19)
[10/12/2017 1:21:34 PM][ERR] SeekUnroll.Test(boxedIndex: 19): Assert.Equal() Failure
Expected: 19
Actual:   159
[10/12/2017 1:21:34 PM][ERR]    at Xunit.Assert.Equal[T](T expected, T actual, IEqualityComparer`1 comparer) in C:\BuildAgent\work\cb37e9acf085d108\src\xunit.assert\Asserts\EqualityAsserts.cs:line 35
   at SeekUnroll.Test(Int32 index, Boolean isXunitBenchmark)
   at SeekUnroll.Test(Object boxedIndex)
[10/12/2017 1:21:34 PM][INF]   SeekUnroll.Test(boxedIndex: 11)
[10/12/2017 1:22:06 PM][INF] Finished 5 tests in 168.196s (2 failed, 0 skipped)

Expected results:

• Tests should pass.

[briansull]

The are no x86 JIT checkins that could have caused this to break:

The last successful run was at
Build #4531 (Oct 11, 2017 10:27:41 AM)

https://ci2.dot.net/job/dotnet_coreclr/job/perf/job/master/job/perf_perflab_Windows_NT_x86_full_opt_legacy_backend_smoketest_prtest/lastSuccessfulBuild/

The first failing run was at
Build #4533 (Oct 11, 2017 11:19:46 AM)

https://ci2.dot.net/job/dotnet_coreclr/job/perf/job/master/job/perf_perflab_Windows_NT_x86_full_opt_legacy_backend_smoketest_prtest/1649/

Some infra-structure change must have taken place around 11 AM on Wednesday that cause this leg to start failing

[jorive]

@briansull The scripts that gather the data to be uploaded do better validation. This issue is not new, we just found it because of the new validation.

[briansull]

From looking at the log it appears that all of the last 9 tests are reported as failing:

12:23:06 Running Perf-Off Deserialize
12:23:12 Running Perf-Off Serialize
12:23:20 Running Perf-Off ConsoleMandel
12:27:51 Running Perf-Off RayTracer
12:27:56 Running Perf-Off SeekUnroll
12:31:46 Running Perf-Off Indexer
12:32:52 Running Perf-Off SpanBench
12:33:03 Running Perf-Off Crypto
12:33:08 Running Perf-Off Richards

[jorive]

Those tests are passing, the initial error cascaded down as we kept appending to the file instead of bailing out.

[BruceForstall]

Closing all x86 legacy backend bugs