How to choose a test automation framework in 2021

Choosing a test automation framework depends on a lot of factors and is different for every company, team and individual.

Testing is an integral part of the software development lifecycle, thanks to which, developers can ensure that their software product fully performs as intended. While manual testing has its use cases and benefits, manually validating each change can quickly become unsustainable, pushing developers and QA analysts towards test automation frameworks. Such frameworks enabling testing in a set, repeatable fashion, ensuring that guidelines and protocols are followed every single time.

By integrating testing into the software development process, developers can quickly get feedback on any changes made. Test results are ready as soon as changes to the codebase are introduced, letting the team know if a change has broken any of their application’s functionality. This feedback cycle allows developers to work faster and with greater confidence.

There are various levels of automated tests that developers can use. The test pyramid provides an approach for selecting which tests to implement by type, encompassing end-to-end (E2E), service, and unit testing:

A basic test pyramid

Its essential point is that you should have many more low-level unit tests than high level service tests running through a GUI. Unit tests in this context not only apply to the backend, but can also be done for UI components on the frontend. On the other hand, service tests focus on the integration between various functions or components, and end-to-end tests validate entire features.

Reasons for using automated tests

If you have never written automated tests, you might be wondering why you should use an automated test framework in the first place.

There can be a few reasons for this. First, test automation lets you cut the amount of time spent on testing while creating a smoother development experience for software engineers.

Next, automated testing allows you to build confidence in your software product and gives you capacity to support more customers and more load than with merely manual testing. By running hundreds of different tests and case scenarios, you can catch issues that would otherwise make it to production. As a result, developers can build better products that are more pleasant to use.

And of course, automated tests allow for more precise results that capture greater detail about each error. An automated test framework will repeatedly execute all of the steps defined in the test in the same order and manner. The framework’s reports will include the details of each test’s result and run time, making it easier to pinpoint any differences between test runs after implementing new fixes or features.

Keep in mind that automated testing cannot always replace manual testing, as in the case of exploratory testing. It’s important to be aware of the uses of both before proceeding to automate certain tests.

Selecting the best test automation frameworks

There’s no single best solution that works for all cases. Developers must ensure that they make the right choice for their particular company, team, and tech stack. While there are many frameworks to choose from, our general recommendation is to pick the easiest option.

Specifically, if your project already uses a test automation framework, consider using the framework that’s currently in place. And if you’re working on a new project, we recommend going with one of your programming language’s standard unit testing frameworks (see the next section for language-specific recommendations).

The options we present below are all open source and can be used across all types of projects. Some options (like Cypress) have additional paid services, but you don’t necessarily need them when simply trying the framework.

Our guide covers unit tests and end-to-end tests. Unit test frameworks are able to conduct API-level tests as well as function-level tests, while end-to-end tests focus on testing an entire feature.

But what about service-level tests? The optimal way to implement this layer of testing will depend on your particular project.

For example, if your service-level test validates the interaction of two back-end services through an API or a client library, we recommend using your unit test framework to write this test.

If your are testing the integration between a back-end component and a UI component, using an end-to-end framework is a better fit.

Best test automation frameworks for unit testing

The following unit testing frameworks are well adopted within their respective ecosystems, and we thus recommend them to developers getting started.

The above tools come with features that will allow you to do most (or all) of the unit testing that you’ll need.

There are other unit test frameworks that offer greater functionality or different syntax, but switching between them usually offers only a small improvement for your workflow.

Best test automation framework for feature-level testing

Cypress Cypress is our recommended option for end-to-end testing in the context of new projects. We recommend Cyprus because it’s:

  1. open-source (paid service available);
  2. easy to get started with; and,
  3. covers most end-to-end needs for new projects.

Cypress focuses on quickly providing developers with feedback. By seeing how your changes affect test results in real-time, you end up spending less time and resources on developing, testing, and subsequently adding fixes for any issues that may arise.

Writing tests with Cypress is straightforward: the framework combines all required tools and ensures a seamless experience between them. Developers don’t need to install any dependencies or additional tools separately. The tests are written in JavaScript, and the Cypress dashboard adds a major advantage by visualising test behavior. You can even look through the history of past test runs and see what changed between them. By increasing the readability of the debugging stage, Cypress makes it easier for developers to quickly identify and fix errors.

Also, any Cypress test failures trigger an automatic screenshot or video, so you can play it back and have a more visual representation of what went wrong.

There are quite a few examples of using Cypress online. Developers can find support within the Cypress community and in numerous online tutorials, including on our [blog]({{<ref "docs/frameworks-and-tools/cypress" >}})!

Selenium WebDriver For existing projects, especially those in the Java ecosystem, Selenium Webdriver is the default standard.

A lot of end-to-end web application testing tools are based on Selenium. This is also the case for WebDriver — a test automation tool for browsers.

WebDriver offers a high level of customisability and flexibility to its users, particularly those with programming proficiency. WebDriver is easy to combine with other tools in your testing stack. Supported test scripts include Python, Ruby, C#, Java, and many more.

Unlike Cypress which runs locally in the browser, WebDriver accesses the web application by driving the browser over the network. This allows it to run tests in any browser, and support for multiple browsers (including older versions) is frequently a requirement for legacy projects.

One of WebDriver’s disadvantages is its complexity. For example, to set up WebDriver, you need to install multiple components and wire them together on your development machine before you are able to start running tests. We don’t recommend WebDriver for new projects given that its not as user-friendly as Cypress. Nonetheless, Selenium has dominated the testing tool market for web applications for a decade, and thus has a large community and can work with almost any tech stack. For legacy projects that need support for older browsers, we think WebDriver is still the best option.

Automate email tests with Mailosaur

Test automation frameworks are crucial to ensuring that a product works perfectly under all circumstances. In this guide, we explored some of the best frameworks for unit and end-to-end testing.

If you’re planning on testing your application’s email or SMS functionality, Mailosaur has got you covered. Our service works with all the frameworks we’ve discussed in this article, including Cypress and Selenium.