Bugs have been part and parcel of computer software since the invention of computers. The term ‘bug’ is often attributed to computer scientist Grace Hopper, who discovered a moth stuck in her computer’s hardware, causing it to malfunction. However, unlike this literal bug, most bugs today are inadvertently embedded in software by programmers themselves.
Bugs sometimes remain invisible and find their way into a released product unless caught by testing procedures. Exploratory testing, one of the types of testing that developers and quality assurance teams can do, helps discover such issues. In combination with automated testing, exploratory testing is indispensable to the software development process.
Read on to learn about exploratory testing, why it’s important in Agile environments, and our take on when exploratory testing is (and is not) the right tool for you to use.
What is exploratory testing?
Testing today is an integral step of software development, ensuring that a product is bug-free and functions as intended. Many developers have a strong preference for the automating tests. In the long run, test automation is often faster and more thorough than manual testing. It also serves as guidance for developers and the QA team alike.
However, automated testing is not an all-purpose testing solution. Errors may arise from the parts of the application that are less thoroughly tested because they are not deemed most crucial to a business’s requirements. Additionally, automated testing may be too expensive or difficult to implement given the expected reward, or it may be too unreliable and produce more false positives or false negatives compared to a human tester. Exploratory testing may prove to be useful in such scenarios.
Exploratory testing is an approach to testing in which a piece of software is tested on the fly, without a pre-existing process. The tester, usually a QA analyst or developer, examines the software by using it in a way that mimics how a real user might use it. In so doing, the tester hopes to identify bugs that would otherwise remain undetected by pre-scripted tests, preventing the bugs from being shipped in the final product.
The fact that it’s a free-form approach does not imply a lack of structure. Of course, it’s impossible to exhaust every single combination of inputs that a user might enter, or to recreate every program state that might follow from a given input. However, the structure of exploratory testing comes from limiting a test’s coverage to only the most important subset of scenarios that haven’t been automated and would instead benefit from exploratory testing.
How does exploratory testing work?
The tester starts by consulting with product owners in order to understand the scope of the releases. In doing so, they come up with an initial testing strategy that covers a wide breadth of scenarios that might reveal a problem. However, given the exploratory nature of this testing approach, such strategies ought to be flexible rather than fixed.
A tester typically comes up with a specific scenario that’s likely to happen to a real user of the application, and then breaks it down into a sequence of events. As the tester executes the sequence step-by-step, they document all faults encountered in the process. Once they’re done, the documentation serves as a blueprint for translating the exploratory test into an automated test for further analysis.
For example, let’s say a software engineer adds a product feature that greets new users upon sign-up. To examine if the feature is working as intended, they might test the following:
- Is the email delivered to the proper recipient and with the appropriate content?
- Do the most popular email services display the email correctly?
A popular testing approach is to try to make the application fail by giving it illegal input. In this case, the tester might want to determine what the email displays for users whose first and last names contain special characters (e.g. from scripts other than Latin), for names that are too long, or for names that contain reserved characters (such as the escape character \n).
Why is exploratory testing important for Agile?
Exploratory testing is about continuously learning, creating and executing tests. This makes it fit in perfectly with the agile methodology where testing requirements tend to change with each development cycle. Since these cycles tend to be short, writing pre-scripted test cases that might become unusable in a few weeks' time when the next cycle comes around is not the best use of one’s time.
In contrast to this, exploratory testing reflects the agile software development manifesto’s tenet of valuing “working software over comprehensive documentation”. It significantly shortens testing cycles because the tester does not need to get bogged down in creating complex documentation if it’s going to become obsolete soon. This reduction in cycle time, in turn, allows the team to quickly respond by creating high-priority scripted tests or to come up with testing plans that they might follow in the future.
When should you conduct exploratory testing?
Exploratory testing is best used in conjunction with other testing methods. For example, it can be used to inform the development team about the areas that require more comprehensive automation, but the reverse holds as well: the results of a scripted test run might point to a part of the application that’s worth further exploration.
This testing approach carries less obvious effects, as well. Although the best exploratory testers owe their success to domain expertise and natural curiosity, any team member with technical skills or knowledge of the product can be involved in the process. Different stakeholders will likely approach exploration from different angles, expanding the scope of the test coverage and possibly exposing bugs that would have otherwise remained hidden.
Consequently, as the testers examine the system and learn about it, they’re able to identify weak spots and provide feedback on improving user experience even if they do not find any obvious faults with the system. Exploratory testing is useful in such scenarios that rely on human insight and are conducted slowly and thoughtfully; fast and routine work is still best done by computers.
When should you not conduct exploratory testing?
Exploratory testing as the sole method of software testing is usually unsustainable in the long run because the lack of a stable and structured testing approach will inevitably allow bugs and failures to go unnoticed. Additionally, exploratory testing requires time and patience and can significantly extend the duration of release cycles, especially when the number of test cases increases over time.
For that reason, it’s almost always worth investing in automated tests as they are much faster and provide consistent results in which you can have greater confidence. For example, if you employ automated testing from the beginning of your product, automated tests will ensure that your product is always thoroughly tested. This proves to be invaluable as complexity and time involvement grows with each additional feature. A guarantee that every release is stable and less time is spent on debugging greatly increases speed to market.
Mailosaur - exploratory and automated email tests
Software testing is a difficult task due to an overabundance of factors that may have different effects on software execution.
In this article, we explained the advantages of exploratory testing. As the tester uses information gained during testing to inform the future course of exploration, they strategically broaden the scope of the areas being tested. This is an effective way to detect edge cases in software that would otherwise remain hidden to structured tests. This approach can be an especially good fit for the agile software development methodology where teams need to quickly react to challenges.
If you’re looking to conduct exploratory and automated testing for email applications, Mailosaur provides an API for automating user signups, password resets, SMS testing, or any other related functionality!