Mobile Test Automation Using AWS Device FarmUtilizing the Java SDK to integrate AWS Device Farm efficiently into a CI pipeline.
Martin SchneiderBlockedUnblockFollowFollowingMay 6Somewhere along our mobile test automation journey most of us reach a point where we consider moving to a cloud platform.
The advantages sound promising: a large variety of devices, no maintenance effort, and flexible scaling.
The reality is slightly more complicated, we must always understand our exact requirements before purchasing a plan with any of these vendors.
While in certain cases building a device lab on-premise can be an effective solution, cloud providers are an exciting and useful tool to take any mobile test automation to the next level.
In this article, we describe how to utilize AWS Device Farm’s public cloud in a way that we can easily integrate into a CI pipeline.
For this example, we use Java, Cucumber, and Appium but the concepts apply to other technologies as well.
Server-side execution modelWhen executing tests, AWS (at least in their public cloud) uses an approach which I call server-side execution.
This means that the test code runs entirely on their infrastructure.
The alternative to this is what providers like Browserstack or Saucelabs use where the test code runs on the user’s machine, called client-side execution, and interacts with the cloud service via a web service.
In general, client-side execution is easier to implement; if you already know how to execute tests locally, the configuration changes you need to make are minimal.
AWS’s approach is a bit trickier as it requires a few extra steps, namely packaging and uploading everything necessary to run your tests to their servers.
This comes with some challenges and pitfalls which I’ll cover in this article.
Executing tests via the web consoleThe easiest way to start running tests on AWS Device Farm is through their web console.
It is a tedious process if you need to do this frequently (we will talk about using their API in just a few moments), but going through the web console is an excellent way to familiarise ourselves with the different steps in the process.
First, let’s log into AWS and open the Device Farm dashboard:Then we create a new project:And create our first run:As we are testing a native Android application (StackOverflow), our first step is to upload its APK file:Next, we need to select the type of test.
We use Appium Java JUnit:Using Maven, we can create a test package as a zip file by following the steps in the documentation.
Don’t get thrown off by the statement that only Java 8 is supported.
We will deal with this later (and of course use Java 12 instead).
In this demo, we use the JustTestLah!.test framework using Java, Cucumber, and Appium (Disclaimer: I’m the author ????).
You can either upload the test package from this link or build the demo yourself using Maven (the zip file will be generated in justtestlah-demos/target):Once you upload the zip to AWS, you will be asked whether to use the standard environment or create a custom environment, we opt for the latter.
The test spec is a YAML file listing a set of shell commands to execute during various stages of the test execution.
This gives us some control over the way tests are executed on AWS.
For now, replace the default spec with the following:I will explain the details of this configuration later in the article, for now, let’s save it and move on to the device selection:AWS uses a concept called device pools.
Each test execution will run all your tests on all devices in the pool.
We just need one device to start with so we create a new device pool with a single Google Pixel:Next, there is a step called Specify device state which we skip for now (keep the defaults).
On the last page, we can review the run.
Let’s reduce the timeout so we don’t use up precious device minutes in case something goes wrong:Finally, we can trigger the test execution:You can see the run on the dashboard.
Click on it and select our test device (Google Pixel) to see live logs and a live-stream of the device (it will take a few minutes to set-up):This is the scenario we are testing:If everything goes well you will receive a success message on the test results page:Side note: Take note of the duration of the test execution.
There is quite a significant overhead for setting up and tearing down the device (and some more from executing the test spec and initializing the WebDriver) that you need to take into account when planning your test executions.
Due to this overhead, AWS is generally much better suited to run larger test suites in a single execution than smaller ones in multiple runs.
To some extent, this is true for all cloud solutions but this kind of overhead is different for different vendors and thus an important selection criterion.
What next?Congratulations!.You successfully executed your first Appium tests on AWS Device Farm.
Let’s look at some of the challenges and see how using a custom test spec can make our lives easier.
Then, we will showcase how to use their API to automate the process of scheduling tests.
Re-upload the test package with every changeYou can re-use test packages previously uploaded to AWS Device Farm.
You only need to package and upload them again if there are changes.
This sounds ok in theory, however, since these packages include everything necessary to run the tests, including not only changes in the framework or its dependencies but also changes in your tests, your test data or your test configuration.
Changing a single parameter (for example, specifying which tests to run) always requires a new test package.
Let’s say your test suite contains 100 tests and you want to distribute them across 10 devices to reduce the overall execution time.
AWS and most other cloud providers don’t support this out of the box.
The naive approach to solving this on Device Farm is to create 10 test packages for each subset of tests, with each configured to execute 10 different tests.
However, this requires packaging and uploading multiple large, almost identical zip files.
Now assume that next time you want to distribute your tests across 20 devices or that you want to execute a slightly different set of tests.
You always need to create and upload new test packages all over again.
This is how we first used Device Farm before they introduced custom environments and it was rather painful.
Continually building and uploading 100+ MB zip files makes a CI pipeline a lot more troublesome than necessary.
Custom environments for the rescueIf you take another look at the custom environment specification above, you can see how we can utilize this feature to remove this restriction.
The base64 encoded information includes our test configuration.
This allows us to customize test runs by uploading a relatively small YAML file (the test spec) rather than re-creating the much larger test package.
We encode the configuration during the creation of the test spec.
When it’s executed we decode it back into a file which is then passed to the execution as a parameter.
For a handful of arguments, this may be overkill, but for a larger set of configuration values, a base64 encoded string proves to be quite a robust and elegant way (it also spares us the headache of escaping special characters in YAML).
JustTestLah!.uses Cucumber, so, for example, we can run different scenarios by passing separate Cucumber tags:orThe test package can be the same for each of the runs, only the test spec changes.
Further improvementsWe can take this approach even further and remove the need for a test package altogether.
Instead, we configure the custom environment to fetch the necessary code and configuration from an external source and build it entirely on the AWS instance.
This allows us to separate our test code from our framework code.
The latter should be reasonably stable, only the tests themselves change more frequently (we fetch them during the test spec execution).
Without smart usage of the test spec, such scenarios would be a headache to implement.
2.
Control the execution environmentIf you execute tests on your infrastructure, you are in full control of the tools you use and their versions.
With AWS Device Farm, it’s not that simple.
The default environment still uses Java 8 (which is four major releases behind the current JDK 12).
Other libraries may have similar issues, but for Java, there’s an easy fix it can be downloaded and installed by adding a handful of lines to the environment configuration:However, this is a workaround and not a solution.
You pay for the device minutes from the time the execution starts and any additional, time-consuming steps in the test spec can add up in the long run.
Executing tests via APINow that we’ve gone through the steps of manually creating a test execution let’s see how we can use the excellent AWS Java SDK (there are SDKs for other programming languages too) to automate the process.
For this, you need to create an AWS user with access to Device Farm and generate an access key and a secret key.
I recommend setting up AWS CLI as well and configure it by calling:This will ask you for the access and secret keys generated before.
As a region, set us-west-2.
Your keys will be stored in your user home (under ~/.
aws/credentials) and the default credentials provider used in the SDK will pick them up from there.
Be careful to keep them as far away from your source code as possible.
Once this is done, let’s have a look at some JAVA code.
The full source can be found on GitHub.
For this demo, we call AWS from inside a JUnit runner class (the interesting bits are inside the run method).
Note, that instead of a device pool we use a concept called device filters to determine the test device to use.
Now, even if the tests run on AWS’ infrastructure, we can wrap them in a local JUnit execution (with some limitations) and execute them just the same way as local Appium tests or using a cloud provider with client-side execution.
If you pull JustTestLah!.from Github you can trigger the same tests we scheduled through the web console before by calling:The only value you need to set in stackoverflow_aws.
properties is the ARN (Amazon Resource Name) of your Device Farm project.
You can easily find this using AWS CLI:Please note, that the path to justtestlah.
properties must be absolute (feel free to contribute to JustTestLah! to improve this).
You can find the demo configs under justtestlah-demos/demos.
Using a symlink ln -s justtestlah-demos/demos /demos all commands work as shown.
Alternatively, modify the path as needed.
You can easily skip uploading a new app package or test package by specifying the ARN of an existing one in the properties file:cloudprovider=awsaws.
projectArn=xxxaws.
appPackageArn=yyyaws.
testPackageArn=zzzIf you wanted to run the same tests on Browserstack, you’d just pass a different configuration (set your Browserstack email and access key in stackoverflow_browserstack.
properties):For local execution (make sure Appium is running, and you have at least one connected Android phone or running emulator) you’d call:As you can see, using this technique the cloud provider for executing the tests is nothing more than a configuration value.
You can seamlessly switch between local executions, AWS Device Farm and Browserstack.
The next level of encapsulation would be to maintain different devices across multiple platforms and let the framework handle everything else, i.
e.
pick the cloud provider for each test which can offer the requested device (and can do it fastest).
If you have any experience with this kind of setup or want to experiment with it please drop me a line.
ConfigurationBesides the device filters, there is some additional configuration you can define for AWS test executions (those were the settings we skipped during the Specify device state on the web console):Maven integrationFor a slightly different way of integrating AWS Device Farm (but using the same Java SDK), you can have a look at my Maven plugin.
It provides goals for each step (uploading app package, uploading test package, scheduling a run etc.
) which allows you to separate them.
For example, you can upload a new app package every time there is a new build and upload a new test package every time there is a change in your test code.
The test execution itself can then be independent of both of these and just re-use the latest (or any specific) app and test packages.
Jenkins pluginThere is also a Jenkins plugin for AWS Device Farm but it is not as flexible as the solutions described in this article.
SummaryIn this write-up, we showcased a PoC-style demo of how to run tests on AWS Device Farm in just about the same way as locally or on client-side execution platforms like Browserstack.
Getting your tests running on AWS Device Farm and especially integrating them into a build process in an efficient way is not a straightforward process (for all but the most basic requirements), and I hope the examples can help others who are facing similar challenges.
Once things are running, AWS is a reliable cloud provider for mobile automation, so it’s worth sticking out and customizing things to your needs.
If you have any questions or comments feel free to contact me.
.