How Much Testing Does Your Mobile Application Need?
Mobile applications are different than web or installed applications and deserve some careful thought to make the best choices about what and how much mobile application testing services are really needed.
There is a long list to consider and surprisingly, you probably don’t need all of those mobile app testing services to achieve your goal.
Mobile application testing services can include:
- Mobile application functional testing
- Mobile application regression testing
- Mobile application cross device testing
- Mobile application test automation
- Mobile application security and penetration testing
- Mobile application performance testing
- Mobile application usability testing
Goals for Mobile Application Testing Services
Here are some questions you can consider to help you determine if one of the types of mobile application testing services I listed will help you meet your needs and achieve your goal.
First, what is your goal for your mobile application?
- How many downloads or users do you want to use it?
- What value will your users receive from using your mobile application?
- How will you know they have received that value?
Now that you have a basic goal for your mobile application, see if the following questions help you chose the best and most appropriate mobile application testing services.
Mobile Application Functional Testing
Does you mobile application contain complex or simple functionality?
Simple functionality with only a few inputs/outputs/screens usually doesn’t need anything other than some basic validation that the inputs/outputs and screens work and flow properly. There are few inputs/outputs/screens so the number of combinations to test will be limited and can be accomplished relatively quickly.
Complex functionality where you have more than just a few inputs, outputs and screens probably needs (and deserves) a more robust testing strategy and plan. In general, you can’t test everything (it takes too long and costs too much) so selecting a smart test strategy will be very important to make sure that all of the “Important Stuff” is tested thoroughly and frequently. Almost any qualified Test Engineer or QA Manager will have a small library of Test Strategies and can help you select the one or two that most closely match your needs and goals.
Mobile Application Regression Testing
Regression Testing helps you make sure that you didn’t break anything that was previously working well. The main goal of regression testing is to insure that you don’t disappoint or anger your mobile application users instead of delighting them with new features and functions.
Mobile applications tend to be updated frequently. One of the benefits of a mobile application is that they tend to be smaller and simpler than other types of applications and can be updated very quickly and easily. This is a tremendous benefit because it allows you to quickly add or change features based on real-time user feedback.
But, being very quick to make changes to your mobile application may also hurt you if you don’t take just a little extra time to complete an effective regression test.
Regression Testing Can Be Challenging
- Familiarity breeds contempt. When you’ve seen and tested your mobile applications features and functions every day for the past six months, it’s easy to let your assumptions and urgency take control. This can result in skipping or overlooking some functionality in your haste to release your new mobile application. Using an experienced and trusted third party for your regression testing will insure that nothing is skipped (or assumed to be OK) during your regression testing. “Once over lightly and quickly” is not a good regression testing strategy!
- Lack of test documentation. Mobile applications tend to be smaller than web applications and good test documentation is usually the first casualty. And no, your mobile application doesn’t necessarily need a full set of ISO compliant test documentation. But, a decent list of test scenarios and use cases can help you complete your regression testing quicker and with a much higher (and measurable) level of quality and customer delight. An experienced third party software test engineer can quickly and easily help you build and maintain these test documents as a natural part of helping you regression test your mobile application.
Regression testing is simple but not always easy if you aren’t an experienced software test engineer. But, getting help from a professional, at least the first couple times, can go a long way to helping you release your mobile application faster, better and cheaper.
Mobile Application Cross Device Testing
Your mobile application might be used on a new iPhone or an Android phone several years old. Almost all mobile devices behave slightly differently and the difference in behaviors between iOS and Android can be significant. If you fail to test your mobile application on a variety of mobile devices, you are inviting a catastrophe.
Cross device testing helps your mobile device safeguard against the following.
Customer loss: Cross testing is vital to ensuring quality assurance across all platforms that your application is on. If everyone was using the same device with default graphics, this wouldn’t be an issue. Users today use a variety of devices across several platforms; cross testing across platforms ensures users from all ends have the same experience. If the user interface on one platform performs poorly, customers will likely leave the app quickly.
Issues that arise from upgrades: Mobile applications are updated very frequently and these updates will eventually cause several bugs. Cross device testing will help you identify bugs and how to deal with them. Failure to recognize this might cause significant bugs that can cause your application to be unusable until it is resolved. This can also lead to compatibility issues after design changes as well.
SEO and additional rankings from Google: For mobile applications, ranking on search engines is crucial to gain traction to your site and mobile app. Since 2015, Google has taken a mobile-first indexing approach. This means that the algorithm for Google search results takes cross device compatibility into account and ranks accordingly. If a competitor to your app is compatible with all platforms and devices, they will significantly outrank you.
There are two basic methods available for cross device testing. You should consider the pros and cons of both before specifying what mobile application testing services to obtain.
Real Mobile Devices and Operating Systems
Real devices provide the best mobile application testing results since you are testing your application on the actual device. But, Real Mobile Devices are expensive to acquire and maintain. Real Mobile Devices are also slightly more time consuming to test on as they require physical handling. Finding a sufficient variety and availability of Real Mobile Devices is a big challenge as new devices and mobile operating systems are released and updated frequently – and this makes the process of building and maintaining a library of mobile devices very expensive.
Mobile Device Emulators
Emulators are usually on-line services where you can load your mobile application on a variety of specific virtual or emulated mobile devices with specific operating systems and versions. Mobile Device Emulators are very handy but have some distinct challenges. Mobile Device Emulators work well for basic functionality but are seldom useful if hardware functionality testing (GPS, Cellular Vs Wifi Networks or Device rotation) is needed to test your mobile application.
Also, mobile device emulators can’t emulate the performance of an app when it receives incoming calls, text messages or other notifications. This is a crucial issue because everyone with mobile devices gets numerous notifications every hour. If an app has to crash each time a notification occurs, this would make it unusable. Additionally, mobile device emulators can’t simulate other aspects like battery issues, touchscreen issues and more. These are some of the most fundamental aspects of using a mobile device.
But, We Think There is a Better Way
The TESTCo Blended XDT (Cross Device Testing) – at TESTCo, we take a blended approach to using Real Mobile Devices and Mobile Device Emulators. We will help you analyze your expected customer device usage and the current market share of each mobile device and operating system and produce a matrix that compares your testing needs against Real and Emulated devices. We use this to build a testing strategy that optimizes for primarily Real Mobile Device Testing with a portion of Emulator Testing so that we can give you the highest confidence that your mobile application will work flawlessly on the devices that your customers use.
The TESTCo Blended approach takes advantage of the benefits of both real mobile devices and emulators while covering the blind spots of each approach. This gives your mobile application testing a comprehensive approach that would simulate what would happen to your application in the real world. This ensures that your mobile application will work seamlessly during its first encounter with users doing normal functions on their phone like using it with low battery, taking calls while using your app and more.
Mobile Application Test Automation
Using test automation for your mobile application can allow your software development team to build and release your software quicker. But, is development speed really a problem and do you want to invest to solve that problem?
Here’s why I ask. Test automation is the process of writing computer code (for the test automation tool to use) to test your computer code (your mobile application). So, you’ll be building another computer program (to test your mobile application) that does test your software faster but is more complex and expensive to build and maintain. Go into this one with your eyes wide open for the ongoing consequences and not just the initial expected benefit of faster testing.
Another issue with “going faster” when building and testing your mobile application is the approval and update process – which is entirely out of your hands! Sure, you can build and test your mobile application faster but you still have the issue of bottlenecks getting your mobile app approved for Apple or Google store and then, even after it has been re-approved and loaded in the Store, your chances that your users will actually download and install the update are small.
So, instead of “going faster”, you might find that “building smarter” provides you with more benefits and fewer costs. Nevertheless, building and running test automation can be valuable for your mobile application in certain situations.
Mobile Application Security and Penetration Testing
Your mobile application probably has some personal and/or confidential information on it. Hackers are penetrating, stealing and selling confidential data from mobile applications and their servers on a daily basis. Can you afford for your mobile application to be hacked and your users mad at you?
Thankfully, most mobile applications have a small number of security and penetration points so testing them is common practice.
The security of your mobile application’s login screen probably isn’t an issue. In fact, most mobile applications are naturally quite secure in their user interface as long as the login function is well built – and most are.
The real challenges with mobile application and security lie in the way your mobile application communicates with the world around it and it’s server. Mobile applications use API calls to send and receive information to and from other users and the server in most cases. The robustness, error checking and reliability of these API calls will determine how “safe and secure” your mobile application will be for your users. Mobile application security and penetration testing evaluates your mobile applications login screen and internal functions as well as your API calls to ensure that your customers’ data is safe and secure.
Penetration testing can be done against different types of code and systems in your mobile application like servers and APIs. Depending on your goals for the pen tests, there are several phases within each pen test to ensure that the security of your application is covered from all angles. After a pen test is done, a formal report will be given to you that covers all the vulnerabilities that were found and how hackers can use them to gain access to your application’s data. In contrast to many systems and processes of your mobile application, penetration testing is largely a manual process. There are testers that overlook the entire process to make sure that your mobile application cannot be breached.
Mobile Application Performance Testing
Your mobile application will need to respond and perform quickly for your customers and prospects to feel confident using your app. Poor or questionable performance can (and does) lead to poor reviews, fewer stars and rapidly decreasing downloads and usage. This probably is not your goal so performance testing may be a risk you want to consider selecting for your mobile application testing services.
There are three critical areas to consider when performance testing you mobile app.
Device Performance – Your mobile application needs to do 2 things simultaneously – respond quickly when in use and play well among the other mobile apps on the device. Testing the first is usually straight forward. Testing the second will require a smart strategy to get accurate results quickly without wasting a lot of time with invalid test setups.
Server/API Performance – Your mobile application will need to communicate with your server to send and retrieve information. In most cases, the server application is separate from your mobile app and interacts with your mobile app through an Application Programming Interface (API). This server app will tend to behave like most server-based applications and is subject to the same performance testing with one exception. The APIs need special attention to a wide variety of performance loading criteria to make sure the mobile apps that depend on them are not needlessly bogged down. Make sure to remember to test the server under load as well – not just the server application!
Network Performance – There are almost too many mobile networks to even list anymore. Mobile devices communicate either via WiFi or the carrier’s mobile data network. In all cases, even Wifi, the network speed and capacity are going to vary wildly. 2G and 3G networks still exist in many places in the world and the continental United States. Many WiFi hotspots are still using the older 802.11(a) and (b) technology. Network performance testing is tricky and it is critical to understand the user’s environment and goals before building a network performance testing plan.
Lastly, all performance testing requires detailed data analysis to clearly identify and truly understand the performance bottlenecks. Performance testing typically produces massive amounts of data that needs to be analyzed to be understood. Performance Test Engineers specialize in this data testing and analysis and have spent many years learning and building their knowledge.
Mobile application performance testing can be tricky to plan and execute so that you get the full value of the work. There is a lot of performance testing that can be done but typically only a few of the tests are truly valuable to any certain situation. The key is to be perfectly clear on the problems you want to solve and the risks you want to reduce. This approach allows your Test Engineers to perform the exact tests you need without wasting time and money on unneeded mobile app performance tests.
To Be Continued
In the coming weeks I will shed light on the five other types of mobile application testing so you can make smarter testing decisions. In the meantime, check out this post if want tips on how to select a mobile application testing company.
If you can’t wait for me to finish this post (hey, I can’t write all day, I have a business to run), call me (888-254-9709), or request a call back using the green button on this page. I’m happy to answer your questions.
