[iOS] Infrastructure is failing to report successful test run

[jkotas]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=89051

Build leg reported

System.Runtime.Tests.WorkItemExecution

Pull Request

dotnet/runtime#78288

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

• Open the failing build above and investigate
• Add a comment explaining your findings

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

1. Add the label "Known Build Error"
2. Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation

{
   "ErrorMessage" : "[TCP tunnel] Xamarin.Hosting: Failed to connect to port",
   "BuildRetry": true
}

Additional information about the issue reported

System.Runtime.Tests are reported as failing. It looks like a failure in the reporting infrastructure. System.Runtime.Tests succeeded according to the net.dot.System.Runtime.Tests.log file:

Tests run: 50259 Passed: 50025 Inconclusive: 0 Failed: 0 Ignored: 133 Skipped: 101
Killing process 95249 as it was cancelled
[TerminateWithSuccess]

Report

Build	Definition	Test	Pull Request
89138	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	dotnet/runtime#78593

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
1	1	1

[jkotas]

Another hit in dotnet/runtime#78593

[premun]

Analysis

I can see that the TCP tunnel between XHarness and the device failed:

[12:46:55] dbug: [TCP tunnel] Xamarin.Hosting: Attempting USB tunnel between the port 56661 on the device and the port 56661 (21981) on the mac: 61
[12:46:55] dbug: [TCP tunnel] Xamarin.Hosting: Failed to connect to port 56661 (21981) on device (error: 61)

When this happens, the app just dumps the testResults.xml into its stdout instead the TCP connection (the stdout is then in the net.dot.System.Runtime.Tests.log log). So while the app did perform the actual unit tests fine, the run was considered an APP_CRASH because the app never talked back.

These TCP problems happen every now and then and this is the issue for them: dotnet/xharness#934
We haven't had the time to handle the TCP flakiness yet. It is unclear whether the flakiness comes from MacOS, mlaunch or the runtime/networking stack.

Mitigation proposal

I can see that we have turned retries off for Apple devices due to a small amount of HW but maybe we can turn them on? We now have many more iPhones in the osx.1200.amd64.iphone.open queue. Maybe we can move onto there and turn on retries? This sounds like we should do it anyway? @steveisok what do you think?

Solution (?)

• Proper solution is to figure out why the TCP connection fails but that will be costly.
• We can also make the TCP tunnel logic smarter so that it understands the connection is flopping.
• We can also consider getting the testResults.xml from stdout always but that will be quite problematic because we get the logs through a quite hacky way. All solutions seem quite costly.

[premun]

I don't think we should create "known issue" for this as it might cover potential problems. I think enabling retries is the best next step. I will add open a PR for that. I checked the telemetry and there are not many APP_CRASH results (just 3 for iPhones in the last week or so).

Regardless.. seems like we are not using the new iPhones though - there are 50 iPhones with no work - we should also enable tests for iPhones too probably, not just AppleTVs?

[steveisok]

I don't think we should create "known issue" for this as it might cover potential problems. I think enabling retries is the best next step. I will add open a PR for that. I checked the telemetry and there are not many APP_CRASH results (just 3 for iPhones in the last week or so).

Regardless.. seems like we are not using the new iPhones though - there are 50 iPhones with no work - we should also enable tests for iPhones too probably, not just AppleTVs?

We can enable retries and see if they help. I thought we added iphones to the rolling build. I'll definitely do that if it's not currently.

[premun]

We have agreed that we could make XHarness recognize TCP issues and return some extra exit code and then have a known build error for that case.

[jkotas]

We have agreed that we could make XHarness recognize TCP issues and return some extra exit code and then have a known build error for that case.

+1

I believe that the logic you have added in #11689 is going to hide intermittent test crashes that is bad for getting a signal about the product reliability. Can it be deleted as part of fixing this?

[premun]

I believe that the logic you have added in #11689 is going to hide intermittent test crashes that is bad for getting a signal about the product reliability. Can it be deleted as part of fixing this?

Yes, this is a good point. Let's start distinguishing TCP-caused failures and let the other blow up.
(there have been 12 APP_CRASH results in the last 3 days btw)

I will include this in the issue that I will open for the TCP exit code.

[premun]

Removing Known Build Error as this log line might also appear on runs with unrelated issues.