If you’re looking to create a mobile application, you’re probably already thinking about testing every possible scenario you can think of. Areas such as password resets, button presses, and link verification should all be checked before release. Most programmers benefit from using automated testing to run all such scenarios as efficiently as possible.
In this article, we take a look at what mobile app test automation is, before giving our take on the best automation frameworks to get that testing done.
What is mobile app test automation?
As the name implies, mobile app test automation uses automated testing to root out issues with mobile applications.
To perform test automation for mobile apps, mobile engineers must first choose a test automation framework. This framework is the program that links up to a simulator, emulator, or actual mobile device and handles the execution of automated tests.
Within this framework, it is your responsibility as a software or QA engineer to create test scripts for each test you wish to run. The framework itself will perform the tests so you don’t have to, and will record the results of each test.
The beauty of mobile app test automation is that it simplifies all complex tests and makes it easy to see what works and what doesn’t. In addition, test automation is essential for keeping your QA team productive while increasing the number of features in your application.
Our framework selection criteria
We used the following criteria to help us determine the best frameworks for automated testing on mobile apps:
The framework must be free to download and use
The benefits of free-to-use software are clear. Mobile app testing can be a costly venture, and money saved with a free framework is better spent on other areas of mobile development.
This also means you can get started without having to get on a sales call, or without needing to get a purchase order from your accounting team.
The framework must be open-source or created by the same company as the operating system
For a mobile app test automation framework to make this list, the framework must be open-source or at least be created by the same company that produces the specific OS the framework supports.
Most users prefer open source software (OSS) to avoid vendor lock-in; if a commercial product starts to change in ways that you don’t like, you have no recourse. With OSS, you can keep using the software for as long as you like.
OSS tools allow for modifications. If you hit an issue or a bug and know how to fix it, you can work with the OSS project’s maintainers to create a bugfix.
OSS tools also encourage contributions, so if you’re missing any functionality, you can contribute it to the project for everybody’s benefit.
Because you can invest your own time and energy into the software, OSS frameworks typically garner a large following of users. These users are passionate about the framework, add valuable content, and often help teach others how to use the framework.
The framework must be actively maintained and support the most recent versions of Android/iOS
A framework does little good if it cannot be used to test apps on the latest versions of mobile operating systems. A framework designed for an older version of an operating system lacks potentially critical features that you’d want to test in newer versions of mobile operating systems.
A regularly updated framework will be able to take full advantage of these updates, giving you the best experience possible. Also, seeing regular updates gives confidence that the developer will continue to do so.
Top 3 mobile app test automation frameworks
Out of the handful of available automation frameworks for testing mobile applications, we’ve compiled a list of the three that stand out.
Appium
Appium is an automated open-source framework launched in 2014 for testing native, mobile web, and hybrid applications on both Android and iOS devices. The framework actually allows you to create tests for those platforms within the same API.
Appium itself translates your test commands to XCUITest Driver actions for iOS and the UiAutomator2 Driver actions for Android (both of which we cover below), so it’s possible to use one script for both device types.
To provide even more flexibility, Appium offers client libraries to write tests in several different languages. To cover a wide range of programmers, you can create tests in languages like JavaScript, Python, C#, Ruby, and Swift.
The framework is entirely free to use with all of its features and code available. The development team actively supports the Appium software and keeps it up to date with operating system changes and updates. Sites like GitHub have a very active community of Appium users who constantly work to help each other out.
Appium’s drawbacks are fairly minimal. Since Appium translates code into driver actions during each test, the framework can run more slowly than tests written natively with XCUITest or UiAutomator2. The massive library can make locating commands a bit difficult.
XCUITest
XCUITest is the official Apple-provided automated test framework and is designed specifically for testing apps on iOS devices.
With a razor-like focus on testing the user interface in iOS, XCUITest makes it easy to create tests that reveal the user’s interaction with the app. As a result of this narrow approach, the framework runs quickly and reliably.
Compared to a large framework like Appium, XCUITest is easy to set up and execute, thanks to this focus. However, there are limitations in script writing, as test cases can only be written in Objective-C or Swift. Those programming iOS native applications already know Objective-C or Swift, but if you’re writing a cross-platform app in a framework like React Native, learning iOS-specific programming languages can be a challenge.
XCUITest is free to use and has been so since its debut back in 2015. Even with several years under its belt, the framework still sees regular updates to stay on top of security issues while keeping up with the latest Apple features.
Espresso / UI Automator 2
In many ways, the Espresso framework is to Android what XCUITest is to iOS. Google rolled out Espresso in 2013 with the purpose of letting users create and execute reliable user interface tests on the Android platform.
Espresso offers a Test Recorder to build test scripts by stating interactions, expectations, and assertions for those who aren’t fluent in a programming language. Espresso supports only the Java and Kotlin programming languages for those who want to create scripts in code.
Espresso is more stable than Appium when it comes to testing, and test execution is much faster. The framework’s API is straightforward, with just a few components to maintain.
It goes without saying that by making our list Espresso is open-source, free, and available for anyone to use or modify as they see fit. Created by Google, the framework receives frequent updates and has a solid community to back it up.
Three other frameworks, and why they didn’t make our list
The following frameworks are worth noting but did not end up making our list:
Calabash
Calabash is a test automation framework for both Android and iOS applications. Instead of relying on programming to create scripts, Calabash instead uses application behavior to test different scenarios.
Unfortunately, Calabash is no longer actively supported and will not see any future updates.
Selendroid
Selendroid is Selenium’s foray into automated tests for mobile devices. Selenium is a popular web browser framework based on JavaScript.
While designed to tackle native, web, and hybrid Android applications, Selendroid is no longer actively maintained.
Xamarin UI.Test
Xamarin is an active free and open-source development platform for building a slew of applications across iOS, Android, and even computer-based operating systems.
Xamarin falls short of our consideration because it presently only works with applications created through Xamarin itself.
Use Mailosaur to weed out issues with email and SMS during automated testing
Regardless of whether you use Appium, XCUITest, or Espresso, any automated testing involving either SMS or email functionality can benefit from Mailosaur. Mailosaur enables you to test end to end functionality like password resets and verification messages.