User testing itself goes by many names, for instance, usability testing or UX testing. We’ll be using all these terms here interchangeably.
If we check a few reputable resources for the definition of user testing, we’ll get these variants:
Usability testing refers to evaluating a product or service by testing it with representative users.
Usability testing is a popular UX observational methodology used by UX researchers to uncover problems and opportunities in designs.
Usability testing is a technique used in user-centered interaction design to evaluate a product by testing it on users.
Simple enough, right? So when we talk about mobile UX user testing, we mean examining mobile apps to see if people can effectively use them to address their needs.
Of course, we test the mobile product, not the person using it during usability tests.
Even though the answer to this question may seem obvious, let’s quickly list everybody who usually participates in UX testing:
Someone from the app’s target audience — the most essential member of any UX testing session. They represent the people who will be using your app once it’s out. So pick them carefully.
The facilitator is often a UX expert who instructs and observes participants while (or before) they work on predefined tasks in a test app.
Obviously, this role needs to remain “constant” as much as possible to avoid skewing test results by influencing test users.
Other stakeholders may include QA engineers, developers, or anyone on your team with a strong opinion about certain app aspects who think their questions and observations during the test session will result in more product insights.
Please note that the test user should not be aware of other stakeholders taking part in the test.
Besides the facilitator, representative test users, and (sometimes) other team members, we need the custom built mobile app itself and tasks. What do we mean by tasks?
Well, your app serves a particular purpose, right? Now, you need to offer participants a realistic objective that they can achieve in the app.
For example, suppose it’s a remote patient monitoring app. In that case, you may ask test users to rate their emotions and overall well-being. Therefore, tasks — real-life in-app scenarios (or test cases) are crucial for validating an app’s usability.
Based on where you are with your mobile app at the moment, we can talk about:
This happens when you start fleshing out your app idea by converting product requirements and everything you know about the target audience into wireframes. The beauty of this approach is cost savings because running tests and amending wireframes is way less expensive than straight coding.
MVP is the initial version of your product that will need to verify all your assumptions in the real market. To maximize this effect, you might want to run UX tests right before you’re ready to release the app.
As a result, you’ll be able to adjust onboarding and other minor details that add up to a huge effect in a live mobile app.
Once we launch the app, we usually start gathering its usage data, and if the data shows potential roadblocks in user flows, we need to verify our hypotheses and remove them.
Ensuring usability is critical, and so is working with a team that understands custom mobile software development for user-centric solutions.
Depending on your focus during app examination, you may be after numbers or stories, correspondingly running quantitative or qualitative tests.
Quantitative UX tests are all about numbers and app metrics, for instance:
You run these UX tests to compute approximate cost savings from updating the application UX design, compare with other design versions, or set the future benchmark to compare to.
Qualitative user testing is more common and has to do with discovering inefficiencies in the mobile UX:
Qualitative tests are perfect for uncovering bottlenecks in mobile user flows.
Why would we run a remote mobile usability test on a mobile app? First comes the cost.
Of course, when we initiate application usability testing before coding, we save ridiculously. However, if we engage in mobile user interface testing when the app is already live, we can still save the budget because we won’t be experimenting with the existing user base.
Otherwise, if our assumptions are wrong, we could cause higher user churn.
Getting feedback from test users faster also means that we can release the app into app stores sooner. In a startup world, that could mean the difference between success and failure.
If you think about it, there are not so many options for arranging mobile app usability tests:
If you combine all these options, you get this matrix of mobile usability testing methods:
Even if we weren’t in the middle of the pandemic, I’d suggest that you ditch in-person user tests. Simply because no one behaves the same when they are watched or left alone.
And that has nothing to do with letting participants relax. The sheer presence of a facilitator nearby will skew test results. In-person app analysis is OK:
In real life, especially if you’re a startup, you’re better off running user tests remotely by using special tools. Fortunately, COVID-19 favors this user testing method, too — there’s no need to risk an infection when the whole thing can be easily done remotely.
We’ll speak about the tools you can use for that in a bit, but here’s a brief explanation of how these remote methods really work for mobile app user testing.
Remote moderated user testing is like being on a Skype call with a participant and seeing their screen and interactions in real-time.
During such mobile user tests, besides watching users complete tasks, you also get to talk with them, ask additional questions, or set new tasks. In addition, you can get feedback from other stakeholders who, depending on the tool, may request further clarifications from test users.
Remote unmoderated user testing is the same but without you attending tests. Therefore, such tests can elicit any spontaneous feedback from participants. What you get is a recording and some metrics that participants have generated while completing tasks.
So the chances are you’ll be running remote UX tests on your mobile app. How do you organize that? What steps do you take to run a successful user test that gets you better user traction with your mobile app?
Regardless of where you’re with your mobile app — putting together wireframes or getting downloads from mobile stores — you plan to run mobile usability testing for a reason. Therefore, goal setting is critical.
In other words, you need to know beforehand what you’re looking for when you prepare tasks for users. When creating such tasks for usability testing on mobile phones, we often target:
And of course, setting tasks that follow real-life scenarios for using your app certainly helps to get mobile user experience testing right.
One other thing to note when planning user interface testing for mobile applications is participant selection . We should carefully pick test users based on demographics criteria that we can glean from the app owner: their age, income, gender, location.
The more sophisticated the tool you’re using for remote UX tests is, the more fine-grained filters you can get to pick the right audience for testing the mobile app’s UX/UI.
Testing is obviously a crucial part of usability testing for apps, and the best part about it — you don’t have to attend. As we’ve already established, remote mobile user testing can be moderated and unmoderated.
As you go through the second step, you get to see their pros and cons.
As you can see, with remote moderated UX tests, you get higher interactivity with testers, but this method is more expensive and requires the coordination of test sessions.
As for remote unmoderated usability tests, you can run them faster and on a more tangible budget. However, you won’t be able to ask follow-up questions and therefore wouldn’t be able to glimpse behind a test’s face value.
What’s reassuring is we get to analyze participants’ body language, which is a dead giveaway about their experience with our mobile app. I would argue that unmoderated monitoring works best because there’s no additional pressure on test users.
Finally, once tests are completed, you need to analyze the results. What’s there to analyze, really? Well, you’ll be getting one of these or a mixture of these artifacts:
When analyzing a recording, we’re looking for visual cues. Did the participant run into any issues? Did they spend the same kind of time on tasks we expected? As you can see, analyzing user testing feedback is akin to answering questions that stem from your mobile app research goals.
What’s important is that all this analysis needs to transform into a mobile usability testing checklist with actionable insights. We can then apply these to our mobile app’s UX/UI to improve its overall usability.
Since I’m trying to convince you of the effectiveness of remote mobile user testing, let’s review the most popular usability testing tools for mobile apps that serve the purpose.
We’d want to pick a platform that allows us to cherry-pick test users based on our preferences and collect quantitative and qualitative data. With that in mind, let’s dive in.
Founded : 2009.
Description : Userlytics lets you run remote user experience (UX) tests for any type of applications, including mobile apps.
Highlights : annotations, highlight reels, audio transcriptions, a branching logic for tasks.
Founded : 2013
Description : UXCam is a remote check-up tool that provides video playbacks, screen UX analysis, and user behavior analysis.
Highlights : besides offering session heatmaps and session replays, UXCam
also uses machine learning to identify users that were frustrated with your app during the test.
Founded : 2007
Description : We love and use UserTesting all the time, so we might be biased here. The tool provides us with everything we need to uncover potential UX issues in a mobile app and create a winning UX/UI strategy for your product.
Highlights : We appreciate their massive user base that allows us to drill down to perfect representatives of an app’s target audience, the ease of use, and they have everything else you’d expect from a remote user testing service like video recording with notes, etc.
Founded : 2019
Description : UXtweak offers powerful research tools for improving usability of websites and mobile apps, from prototypes to production . Their offering includes content organization, various tests and surveys, and a number of recruiting options.
Highlights : The platforms aims to automate and streamline the whole process of user testing, including research and reporting.
It’s worth noting that when you do user testing prior to coding the app, you’ll work with such tools as Invision, UXPin, Proto.io, or similar. Their purpose is to quickly put together app designs and turn that into an interactive prototype of an app.
Once you have such a prototype ready, test users can click it through in a desktop browser or right on mobile devices.
create a prototype " width="650" height="360" />
We prefer Invision because of its advanced features:
UXPin also has its advantages. They offer prototyping that is accompanied by coding. As a result, we get prototypes that look closer to real mobile apps, but we also have to sacrifice the cost and speed. Coding the logic layer (in UXPin) takes time and costs more compared to designing screens and interactions in Invision.
When we shift our attention to mobile sites and Progressive Web Apps (PWAs), it’s important to note that the landscape of testing tools has been changing. Unfortunately, Google discontinued its Mobile-Friendly Test tool as of December 2023, which was a blow to many developers.
However, all is not lost. Developers can still utilize Google’s Lighthouse tool, an open-source automated tool aimed at enhancing the quality of web pages, especially when building PWAs. The value of Lighthouse lies in its ability to provide actionable insights about your PWA’s performance, accessibility, and more.
Simultaneously, another useful resource that developers can leverage for bulk mobile-friendly testing is offered by EXPERTE. This free alternative tool provides comprehensive reports, helpful for identifying and addressing any potential usability issues.
So, while Google’s Mobile-Friendly Test tool may be out of the picture, there are still robust options available to developers. Remember, consistent usability testing is a critical part of mobile web and PWA development, helping ensure a seamless and satisfying user experience.
If you’re absolutely bootstrapped, you can ask your friends and relatives to use iPhone’s built-in screen recording option. That way, you’ll get videos of them testing your app. Alternatively, buy Mr. Tappy’s toolkit for $349 and set up a local user testing lab using this equipment. Finally, you can try screen sharing via Skype or Zoom on a mobile device.
Mobile usability user testing is still relatively new, and not every company does that. So don’t be surprised if the process and tools look a little intimidating. That’s exactly why many app owners miss out on user testing. Still, if you follow the procedure, just know how to address the most challenging parts.
We want our app prototype to appear on the participant’s device without cut-off parts. For instance, to get that right with an Invision clickable prototype, the participant would need to install their app. The key is to avoid testing a prototype in a mobile browser.
Remember that you don’t need to attend remote mobile UX testing and instead review video recordings after tests have been completed.
Some tools would even add timestamps for the most interesting interactions, e.g., when the user encountered a problem in the app.
Many of the health and fintech apps we build target specific, niche audiences. And we’ve discovered that one of the best solutions to quickly ramp up a necessary test audience is to use tools like usertesting.com.
We include mobile UX user testing as part of our Rapid Prototyping Package, ranging between $15,000 – $20,000. However, the outcome is a little more than a thoroughly user-tested mobile app prototype — it’s something tangible you can take straight to investors.
If you’re conducting user testing on your own, reach out to one of the platforms we’ve outlined for pricing. Usertesting is around $20K per year. On the other hand, Userlytics allows you to start with $49 per a 15-minute test session. However that quickly becomes a mess if you want more features, longer sessions, etc. Investing in user testing will be key to lower app development costs in the long run since you’ll know exactly what features your users want, and wouldn’t go overboard only to realize you invested in a feature they’d never use.
After helping startups raise over $165 million, we’ve collected some best practices for arranging user testing for mobile apps. I’d like to share a few of them here with you:
Even if there was no pandemic, testing mobile apps in-person requires too much preparation (read costs more). And you can’t deny the fact that the sheer presence of a facilitator changes participants’ behavior.
Some companies tout special protocols that stimulate participants to voice all their actions in real-time as they navigate around a mobile app.
If you try this technique, you’ll discover it’s nearly impossible to concentrate on a task and, at the same time, explain what runs through your mind at the moment.
Once the tests are over, and it’s time to analyze findings, remember to get a second opinion from your colleagues to make sure you aren’t making false assumptions. Otherwise, you might end up with unnecessary app modifications, which brings you back to another round of user testing.
Book one-two extra meetings to compensate for potential no-shows. That way, you still get the results in a timely manner, even if some participants couldn’t make it.
Nielsen Norman Group recommends recruiting 5 users, which should cover most mobile app owners’ needs. However, if your mobile product is robust (over 30-40 screens), you will benefit from running usability tests on 15 participants in 3 successive batches. That’s enough to uncover 99% of the UX issues in the mobile app.
Most remote user testing tools will require SDK installation. So pick the one that won’t slow down your app. We haven’t tested them all, but we can vouch for UserTesting.
You’re kind of already addressing this when you run mobile UX tests remotely because all participants get the same task description. Still, if you’re participating and have spontaneous follow-up questions, make sure you voice them with all the remaining participants.
A word of advice about tasks themselves: try to make them as detailed as possible, including all “presets” that users need to follow, e.g., device orientation, sound on/mute, cellular connectivity, etc.
Avoid giving participants additional time at the start to familiarize themselves with the app. That’s a perfect scenario for ruining their first impression of the app, including onboarding and figuring out the app’s basics.
Run a sample with 1-2 users to ensure your instructions are clear and everything works as expected. You might find out that better definitions will help you speed up the user testing application process.
If you have more questions about mobile app user testing, reach out to speak with our consultants .
