`os_version` invalid error when trying to test iOS 13

[mgol]

Expected Behavior

I'm trying to test iOS 11, 12 & 13; they should all spawn.

Actual Behavior

iOS 13 doesn't spawn.

Steps to Reproduce the Problem

1. Provide the following browsers field in the config:

  "browsers": [
  	{
  		"os": "iOS",
  		"os_version": "11",
  		"device": "iPhone X"
  	},
  	{
  		"os": "iOS",
  		"os_version": "12",
  		"device": "iPhone XS"
  	},
  	{
  		"os": "iOS",
  		"os_version": "13",
  		"device": "iPhone 11"
  	}
  ]

2. Run browserstack-runner.

browserstack.json

Platform details

1. browserstack-runner version: 0.9.1
2. node version: v12.16.3
3. os type and version: macOS 10.15.4 (19E287)

Details

With the above config I'm getting an error:

Error from BrowserStack:  Error: Validation Failed - `os_version` invalid

iOS 11 & 12 run then tests correctly, iOS 13 doesn't. I'm providing correct os_version & device according to the v4 API.

[mgol]

I have many more issues, actually. This one at least shows an error, although it'd be much easier to follow up if it showed the browser entry that's problematic. However, entries for iOS 10 & IE 9 just don't fire up for me at all. These ones:

browsers: [
		"ie_9",
		{
			"os": "iOS",
			"os_version": "10.3",
			"device": "iPhone 7",
 			"real_mobile": false
 		}
]

I also tried a different iOS 10, same result:

browsers: [
		{
			"os": "iOS",
			"os_version": "10",
			"device": "iPhone 7"
 		}
]

For these browsers, I don't see any error when they get spawned, all I see is that they don't show up in the console and at the end I see a grouped error message; example:

Using config: /Users/mgolebiowski/projects/public/jquery/jquery-color/build/browserstack-legacy.json
Launching 9 worker(s) for 9 run(s).
[Windows 7, Firefox 75.0] Passed: 56 tests, 56 passed, 0 failed, 0 skipped; ran for 571ms
[OS X Mojave, Safari 12.1] Passed: 56 tests, 56 passed, 0 failed, 0 skipped; ran for 349ms
[Windows 8.1, Edge 81.0] Passed: 56 tests, 56 passed, 0 failed, 0 skipped; ran for 420ms
[Windows 8, Chrome 81.0] Passed: 56 tests, 56 passed, 0 failed, 0 skipped; ran for 345ms
[iOS 8.3, iPhone 6] Passed: 56 tests, 56 passed, 0 failed, 0 skipped; ran for 372ms
[iOS 9.3, iPhone 6S] Passed: 56 tests, 56 passed, 0 failed, 0 skipped; ran for 317ms
[Windows 7, Internet Explorer 10.0] Passed: 56 tests, 56 passed, 0 failed, 0 skipped; ran for 372ms
[iOS 7.0, iPhone 5S] Passed: 56 tests, 56 passed, 0 failed, 0 skipped; ran for 335ms
All tests done, failures: 1.

You can see that in the jquery-color repo. Relevant files are in the build directory, you can also run it via npm run browserstack.

[Krinkle]

I'm running into the same issue as well. It seems iOS 11, 13 and 14 are not considered valid, despite being listed at https://www.browserstack.com/automate/browsers.json and https://www.browserstack.com/list-of-browsers-and-platforms/js_testing.

Launching 8 worker(s) for 8 run(s).
Error from BrowserStack:  Error: Validation Failed - `os_version` invalid
    at IncomingMessage.<anonymous> (/qunit/node_modules/browserstack/lib/client.js:73:9)
    at IncomingMessage.emit (events.js:194:15)
    at endReadableNT (_stream_readable.js:1103:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
Error from BrowserStack:  Error: Validation Failed - `device` invalid
    at IncomingMessage.<anonymous> (/qunit/node_modules/browserstack/lib/client.js:73:9)
    at IncomingMessage.emit (events.js:194:15)
    at endReadableNT (_stream_readable.js:1103:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
Error from BrowserStack:  Error: Validation Failed - `os_version` invalid
    at IncomingMessage.<anonymous> (/qunit/node_modules/browserstack/lib/client.js:73:9)
    at IncomingMessage.emit (events.js:194:15)
    at endReadableNT (_stream_readable.js:1103:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
[ios 12] Passed: 225 tests, 220 passed, 0 failed, 5 skipped; ran for 1739ms
[ios 8] Passed: 219 tests, 211 passed, 0 failed, 8 skipped; ran for 1227ms
[ios 7] Passed: 215 tests, 207 passed, 0 failed, 8 skipped; ran for 10025ms
[ios 10] Passed: 223 tests, 218 passed, 0 failed, 5 skipped; ran for 2012ms
[ios 9] Passed: 220 tests, 215 passed, 0 failed, 5 skipped; ran for 4378ms
All tests done, failures: 0.

Config:

  "browsers": [
    {
      "device": null,
      "os": "ios",
      "os_version": "14"
    },
    {
      "device": null,
      "os": "ios",
      "os_version": "13"
    },
    {
      "device": null,
      "os": "ios",
      "os_version": "12"
    },
    {
      "device": "iPhone SE",
      "os": "ios",
      "os_version": "11"
    },
    {
      "device": null,
      "os": "ios",
      "os_version": "10"
    },
    {
      "device": null,
      "os": "ios",
      "os_version": "9"
    },
    {
      "device": null,
      "os": "ios",
      "os_version": "8"
    },
    {
      "device": null,
      "os": "ios",
      "os_version": "7"
    }
  ]

[Krinkle]

Most of the Android versions don't seem to work either. I've tried many variations (with and without .0), but only 4.3 and 5.0 seem to work. versions 6, 7, 8, 9, 10, and 11 all yield:

Error from BrowserStack:  Error: Validation Failed - `os_version` invalid

  "browsers": [
    {
      "device": null,
      "os": "android",
      "os_version": "11.0"
    },
    {
      "device": null,
      "os": "android",
      "os_version": "10.0"
    },
    {
      "device": null,
      "os": "android",
      "os_version": "9.0"
    },
    {
      "device": null,
      "os": "android",
      "os_version": "8.0"
    },
    {
      "device": null,
      "os": "android",
      "os_version": "7"
    },
    {
      "device": null,
      "os": "android",
      "os_version": "6"
    },
    {
      "device": null,
      "os": "android",
      "os_version": "5"
    },
    {
      "device": null,
      "os": "android",
      "os_version": "4.3"
    }
  ]

In the builds overview I see:

• Android 4.3 (Amazon Kindle Fire HDX 7)
• Android 5.0 (Null)

And the others never show up.

[Namrata-m13]

@Krinkle I believe os_version needs to be specified as '11.0' or '12.0' instead of '11' or '12' both on iOS as well as Android devices.
However, to avoid these errors, you can also configure your tests exactly as per the configuration mentioned here, https://github.com/karma-runner/karma-browserstack-launcher#configuration

[mgol]

#244 should have fixed the issue (though I haven't tested it personally). Closing.