Parallel testing: get feedback earlier, release faster

We will tackle parallel testing on physical devices later in the article.Benefits:Simple, intuitive configurationFaster test executionFaster feedbackAllows for more releasesCheck the application server on multiple devices at the same time (request, timeouts etc.)No need to create further test targets, to distribute test classes between themGreat integration with CIDrawbacks:In the case of an unstable back-end / application server the possibility of occurring timeouts (several requests at the same time may cause longer responses)Plenty of processing power needed to keep simulators stableFlakiness — if any of the above problems occurCloning the same device / system configuration, not being able to test at the same time on several different configurations (we must use the command line)Assumptions:The basic condition that must be met to use parallel testing is the independence of test cases..For example:We cannot rely on test B starting only after test A is completeA faster execution or result of test B cannot affect the results of test A and vice versaUseful parameters:maximum-concurrent-test-simulator-destinations NUMBER the maximum number of simulator destinations to test on concurrentlyparallel-testing-enabled YES|NO overrides the per-target setting in the schemeparallel-testing-worker-count NUMBER the exact number of test runners that will be spawned during parallel testingmaximum-parallel-testing-workers NUMBER the maximum number of test runners that will be spawned during parallel testingEffective division of testsTo maximise the advantages of parallel testing, we must divide the test cases into classes..Let’s use our international money transfer application, Azimo, as an example, and assume that our full test cycle contains five test cases.The first option, which we definitely don’t recommend, is to create one large class containing all five test cases:It is far better to create several smaller test classes:In the first example, we have no way to test in parallel because Xcode assigns test classes to simulators: namely one free test class is allocated to one free device..Even if we have five devices available, only one can be used..In the second case, each class can be tested on a separate device.If you have more classes than available devices, say six classes and only five devices, the first device that completes testing will receive the sixth and final class.Run in parallel via command lineIf we want to test in parallel on physical devices, we are forced to use the command line..There are several ways to do this.For parallel testing using the command line, we need the ID of our devices..The first step is to display all connected devices and simulators together with their ID, using the following command:xcrun instruments -s devicesUsing the ID of a specific physical device, we can run the command responsible for running parallel testing.In the project folder, we run the following command:xcodebuild -scheme Azimo -destination id={deviceID1} -destination id={deviceID2} testThis command will run the same test classes in parallel on two different devices..This approach allows us to test on different devices and systems simultaneously, but doesn’t save us any time.To speed things up, we can manually separate classes into devices..First we need to build our testing project with the following command:xcodebuild -project Azimo.xcodeproj -scheme Azimo -destination id={deviceID} build-for-testingThen we divide our test classes into the number of available devices..For example, with four test classes and two available devices, it is best to divide the two test classes into each device as follows:xcodebuild -scheme Azimo -destination id={deviceID1} -only-testing: Azimo/login_tests -only-testing: Azimo/register_tests est-without-building &xcodebuild -scheme Azimo -destination id={deviceID2} -only-testing: Azimo/createRecipient_tests -only-testing: Azimo/createTransfer_tests test-without-building &In the case mentioned above, separating the classes into devices saves a lot of time..The optimal solution would be to use four devices, one for each class, but this isn’t always possible.In the case mentioned above, separating the classes into devices saves a lot of time..The optimal solution would be to use four devices, one for each class, but this isn’t always possible.Another way to divide test classes is to create a new test target, eg Azimo1, and to include there appropriate test classes other than in the Azimo Target.Adding classes to the test targetAssuming that the Azimo target has different test classes than the Azimo1 target and vice versa, we run parallel testing on both targets on different devices by using the following command:xcodebuild -scheme Azimo -destination id={deviceID1} test-without-building & xcodebuild -scheme Azimo1 -destination id={deviceID2} test-without-building &The test execution time will be equal to the execution time using the -only-testing parameter together with the determination of the test classes.As you can see, the above methods work but require some human effort (creating new targets, separating tests between targets, maintaining targets when adding new cases / test classes).. More details

Leave a Reply