Testing is integral to any production process. In the context of software design and development, testing provides all stakeholders with confidence that a product is able to provide the functionality defined at the start of the project and meet all non-functional and technical requirements.
The test pyramid is a useful tool for developers and QA engineers who are getting started with testing. Visualising the various layers of software testing allows these professionals to best focus their testing effort.
In this article, we provide an overview of the test pyramid, as well as how to interpret and use it to generate useful takeaways in the context of your software project. Let’s get started!
What is the test pyramid?
Generally speaking, a test pyramid gives developers and QA engineers an idea of how many and which tests they should perform on a given product. The purpose is to encourage critical reflection about the layered nature of software testing. In this guide, we use the test pyramid version that was popularised by software industry thought leader Martin Fowler:
From this simple graphic, we see that an automated testing effort may be divided into unit, service, and end-to-end tests. In some cases you may encounter manual testing depicted at the top, typically in a cloud. The idea is that as you rise from the base of the pyramid to the top, your tests become longer and more expensive to run and maintain. Meanwhile, base tests (aka unit tests) should be quick to perform and numerous.
Of course, your pyramid’s final structure may differ from project to project.
Developing your own test pyramid
It’s important to note that the test pyramid is there not to restrict but rather to guide your testing effort.
As a developer, you’ll need to choose how to write your tests and where to focus your time. This decision (or series of decisions) can ultimately impact production quality and completion time, as focusing on too many end-to-end or manual tests could extend production time and produce less confidence in a final product since you may not end up covering the entirety of the code.
By visualising your testing approach, you can communicate your proposed testing approach to your organisation. The pyramid as a shape is a simple way to display the balance between the different tests.
The best practice test pyramid
Let’s dive into what would go into each type of test in a best practice pyramid.
At the pyramid base, unit tests ensure the proper performance of individual functions. This includes both the function being able to fulfil its regular tasks and handle edge cases as expected. It’s important to have many detailed unit tests. Thankfully, these are short, cost-effective, and easy to run.
Next, service tests or integration tests check the interaction between different functionality or components of the product or program. Whether it’s to check a product’s ability to access a database or to see if an integrated API is working correctly, such interactions are crucial to the product running without issue. At this level, executing a test requires multiple services or components to be running simultaneously, so each test run is slower and requires more compute time compared to a unit test.
At the pyramid’s top, end-to-end tests usually involve the largest number of components all running together. For example, testing a relatively complex web app end-to-end could require running a web browser (with test automation), a web server, an application server, a database, and any supporting services. The more components that need to be run, the more costly a test is to execute. Moreover, with complex end-to-end tests it’s common to run into ‘test flakiness’ as a test intermittently passes or fails due to reasons unrelated to the code being tested, such as when browser automation fails to correctly kick in.
The main advantage of following this testing distribution is the ability to get good coverage of every aspect of the product by catching most issues quickly. Quick action that starts in simple unit tests can prevent large losses further on. This test pyramid shape also contributes to fast execution and product delivery while providing confidence in the result.
Comparing your test pyramid to the best practice version
If you were to take a look at your current test pyramid, you might notice that it does not bear the same shape as the best practice version. And that’s okay! All projects are different.
But if you wish to reap the benefits we mentioned earlier, here’s what you can do:
Step 1. Measure the shape of your pyramid
Before you can start implementing the best practice shape of the test pyramid, you should first determine your pyramid’s current state.
One way to measure the approximate shape of your pyramid is to calculate the test coverage on each layer within your product. If 80% of your code is covered by unit tests, 40% by integration tests, and 10% by end-to-end tests, that gives you a rough estimate of each pyramid’s layer. There is no need to calculate coverage manually; you can take advantage of automated tools like, for example, Istanbul in the JavaScript ecosystem or Coverage.py if you are programming in Python.
Alternatively, you can calculate the number of different types of test cases in relation to one another. For example, you could have 1000 unit tests, 60 integration tests, and 2 end-to-end tests—although it might be worth adjusting for scale in that case!
The process should also account for manual testing. Make sure you have a method of documenting the entire process to determine the amount of manual tests you already conduct. Oftentimes, companies are unaware of how many manual tests they have and how exactly these are evaluated.
Also take note of any overlapping tests. Overlap between automated and manual tests is both permitted and encouraged; however, you should try to avoid overlap between other layers. If your unit test covers a function, it’s not worthwhile to test that same function in an integration test. Instead, an integration test should focus on how a function interacts with other components in your code.
Step 2. Set up your plan of action
Focus on rules and principles and their incorporation into the core of your production cycle. Merely changing the shape of your testing structure for the sake of having a pyramid does not suffice. Instead, focus on the potential impact of such change to encourage your organisation to develop it into a process.
An example of a well-settled testing process can be seen in debugging. When a tester discovers a bug, a solid action plan will not call for the hasty fixing of that specific bug; rather, the goal is to come up with a way to test for that type of bug to prevent its recurrence.
Another example would be the test-driven development of all new and upcoming features. When using test-driven development (TDD), software engineers write a test that fails before changing the main program. As there is no code yet, the test fails. Then, the software engineer adds code that will make the test pass.
Following the test-driven development practice ensures that your code is written in an easily testable manner. Using TDD also results in a high number of tests that you can continuously run while you’re adding code, to make sure that the changes you’re making don’t affect other parts of your codebase.
People committing to change is the key to success
Clear communication will ensure a successful implementation of the test pyramid within your organisation. Ensure that everyone on your team is clear on the overall goal, whether that’s creating clear unit tests as soon as a new function is added or developing a method for recording manual testing results.
Clear and visible reporting on all changes being implemented will pave the way for commitment from your team members.
Get started with Mailosaur
Even after you create your test pyramid, you might find that some areas are still left uncovered. This might include sign-up flows and validation of email addresses, or end-to-end and UI tests, for which the manual testing process is long.
To automate your testing of these areas, Mailosaur is here to help! We’ll help you quickly and easily start testing account verification emails, password resets, and more.