Tutorial: How to Automate Testing of Two-Factor Authentication with JMeter and Mailosaur

Apache JMeter is a cross-platform application for performance testing and testing the functionality of software. It can be used with Mailosaur to automate the testing of [SMS](/ sms-testing) and email, including two-factor authentication (2FA) and multi-factor authentication (MFA) flows.

Mailosaur gives you real email addresses and phone numbers, as well as APIs and libraries to read from them. This allows you to integrate it with JMeter to automate the testing and validation of password reset links, 2FA codes, and any other notifications resulting from user activity.

Here’s how it’s done.

The tools you need to automate SMS testing with Apache JMeter

If your 2FA or MFA codes or authenticator app don’t work, your users can’t log in to your app or service - a big problem for any development team. This is why it’s important to make sure that authentication codes are reliably received via SMS using real phone numbers.

Mailosaur gives you real phone numbers to test with, as well as email addresses, and also simulates authenticator apps. Once your app has sent a notification, you can use Mailosaur’s APIs or libraries to check whether it was successfully delivered, read its contents, and retrieve other useful metadata. This allows you to test all of your notifications, from authentication to order confirmations and the other transactional emails that your users expect to receive.

Apache JMeter is an open-source tool primarily intended for load testing that can also be used to test the functionality of your code. As it’s Java-based, it can be used with Mailosaur through Mailosaur’s Java library.

This lets you use JMeter for testing notifications or add this kind of testing to your existing performance tests. To do this, you’ll need:

• A Mailosaur account and a test phone number. Start using Mailosaur by signing up for an account.
• A server-restricted Mailosaur API key and its Server ID for testing SMS. You can create this from the Mailosaur dashboard. To generate authenticator app codes, you’ll also need to create a standard API key.
• A mobile device to send an SMS to the Mailosaur test phone number. You’ll need to send an SMS with a test one-time password (OTP).
• Apache JMeter installed on your system. Download JMeter (requires Java 8+). This tutorial was tested using JMeter 5.6.3.

How to automate SMS 2FA/MFA testing using Mailosaur and JMeter

Once you’ve installed JMeter, you need to download the libraries required to run the tests. This includes the Mailosaur Java library and its dependencies (the specific versions of the .JAR Java package files used in this tutorial are noted after each):

• The Mailosaur JAR (mailosaur-java-8.3.0.jar)
• Google Guava (guava-33.4.8-jre.jar) - make sure you download the JRE, not the Android version
• Google HTTP Client Gson (google-http-client-gson-1.47.1.jar)
• Google Gson (gson-2.9.1.jar) from https://repo1.maven.org/maven2/com/google/code/gson/gson/
• Google HTTP Client (google-http-client-1.47.1.jar)
• OpenCensus API (opencensus-api-0.24.0.jar) (specifically 0.24.0, as later versions are missing or have changed paths for TextFormatter, which is a dependency)
• OpenCensus HTTP Utils (opencensus-contrib-http-util-0.31.1.jar)
• GPRC Core (grpc-core-1.9.1.jar)
• GPRC API (grpc-api-1.74.0.jar)
• GPRC-Context (grpc-context-1.9.1.jar)

Copy these JAR files to ./lib/ in the extracted JMeter app directory. (Do not copy to ./lib/ext/, as things won’t work correctly.) You can also integrate your JMeter testing with Maven and use that to manage dependencies.

Next, create a file named jmeterSecrets.properties to contain the API keys and variables for your Mailosaur 2FA Test. The mailosaur.apiKey (the server-restricted API key should be used wherever possible instead of the standard API key to limit over-permissioning), mailosaur.serverId, and mailosaur.phoneNumber values should be taken from your Mailosaur dashboard.

mailosaur.apiKey=xxx
mailosaur.serverId=xxx
mailosaur.phoneNumber=xxx

You can save this file anywhere on your local system where JMeter can read it or create it from a secure store as part of your CI/CD pipeline so that it’s not committed to version control. (Just make sure your execution environment is ephemeral!) Alternatively, you can use environment variables in your CI/CD environment.

Run the following command in the JMeter bin directory to launch JMeter with the secrets loaded as properties:

• Windows: ./jmeter.bat -q .\path\to\jmeterSecrets.properties
• Linux/macOS: ./jmeter -q ./path/to/jmeterSecrets.properties

JMeter should now be running with all of the packages loaded and your secrets available via props.get('property.name'), as demonstrated later in the examples in this tutorial.

In JMeter, a Test Plan defines all of the steps that will be taken during execution. Create a new Test Plan file by selecting File > New, then save it in the location of your choice by selecting File > Save Test Plan as.

A Thread Group manages the execution of your tests. Create a new Thread Group by right-clicking on Test Plan in the JMeter sidebar and then selecting Add > Threads (Users) > Thread Group.

Next, you need to create a Sampler within the Thread Group to run your test code. Right-click on your new Thread Group and then select Add > Sampler > JSR223 Sampler. Rename the new Sampler to Mailosaur OTP Test, and ensure the new Sampler Language dropdown is set to groovy.

Add this code to the new Mailosaur OTP Test Sampler:

import com.mailosaur.MailosaurClient
import com.mailosaur.models.MessageSearchParams
import com.mailosaur.models.SearchCriteria

String mailosaurApiKey = props.get("mailosaur.apiKey")
String mailosaurServerId = props.get("mailosaur.serverId")
String mailosaurPhoneNumber = props.get("mailosaur.phoneNumber")

MailosaurClient client = new MailosaurClient(mailosaurApiKey)

MessageSearchParams params = new MessageSearchParams().withServer(mailosaurServerId) // Set the Mailosaur server to search
SearchCriteria criteria = new SearchCriteria().withSentTo(mailosaurPhoneNumber) // Search for messages to your Mailosaur test phone number

def message = client.messages().get(params, criteria) // Perform message search
def otp = message.text().codes().get(0).value // Extract and return the code from the first result

log.info("OTP code: " + otp)
vars.put("otp", otp) // This lets you retrieve the OTP as ${otp} in later tests

Save the Test Plan by selecting File > Save. You can now run the test by clicking the green Play button (▶) on the toolbar.

Failures will be thrown as errors. Press the ! icon to view the log/errors, and press the double broom icon to clear the log/results. This test will fail until there’s an SMS containing a 2FA/MFA OTP code received by your Mailosaur SMS number.

Send an SMS that looks like a 2FA/MFA verification message (for example, OTP 12345) to your Mailosaur test phone number. You can confirm its receipt in the Mailosaur dashboard.

Re-run the tests in JMeter, and they should now succeed. Mailosaur will automatically extract the OTP code from messages that look like verification messages, allowing you to confirm the receipt of OTP SMS messages from your JMeter tests and to then validate their contents. Note that by default, Mailosaur will only return messages received in the last hour.

How to automate testing MFA codes from an authenticator app

How to automate testing MFA codes from an authenticator app

Mailosaur can also simulate an authenticator app to generate MFA codes for testing. To do this, you need to enable it in your Mailosaur dashboard.

Then update your jmeterSecrets.properties file to use your standard API key (rather than the server-restricted key that’s limited to SMS on a specific server ID), and add the following line containing your authenticator shared secret (created when you configured your MFA solution) to the end of it:

mailosaur.authenticatorSharedSecret=xxx

Add the following import to the bottom of the import statements at the top of your Mailosaur OTP Test sampler:

import com.mailosaur.models.OtpResult 

Finally, add the following Mailosaur OTP authenticator app test code to the bottom of the Mailosaur OTP Test sampler:

String sharedSecret = props.get("mailosaur.authenticatorSharedSecret")
OtpResult authenticatorCode = client.devices().otp(sharedSecret) // Generate an OTP code using a simulated authenticator app provided by Mailosaur
log.info("OTP authenticator code: " + authenticatorCode.code)
vars.put("otp_authenticator", authenticatorCode.code) // This lets you retrieve the authenticator-generated OTP as ${otp_authenticator} in later tests

Authenticator app codes can now be generated and tested against your authentication system.

Automated testing for SMS and email can significantly improve your products’ user experience and success. But it’s often overlooked, leading to frustrated users not receiving vital 2FA/MFA codes, password reset emails, and other notifications.

If you’d like to learn more about what Mailosaur can do for your testing automation, speak to our sales team, check out our documentation, or start a free trial.