Mobile Testing Strategy

The Difference Between a Mobile Testing Strategy and a Mobile Testing Plan

/in /by

In This Post

 

There is a big difference between a mobile testing strategy and a mobile testing plan. If you know the difference, your mobile software testing will be much easier and more valuable.

There are a number of ways of defining a strategy. One of the definitions that we’ve found useful is:

“Identify assets that have advantages that you can put into coordinated action to achieve your goal”

Or, said another way, list and organize everything you think you’ll need to achieve your mobile testing goal and why you think you need it.

Your goal should be clear enough that you can quickly see what is, and is not, relevant to achieving your goal.

For example, if your mobile testing goal is to release your game every week with new features and fixes, your strategy might be different from a goal to release a zero defect trading app. Already, you can start to see that you might need different Assets for each of those goals.

Your Goal Will Dictate Strategy

One of the first challenges in building your mobile testing strategy will be to clarify and tighten your mobile testing goal. Your goal should clearly show what is and is not important to include in your mobile testing strategy.

The biggest and best tool you can use to uncover gaps or weaknesses in your mobile testing goal will be relentless Critical Thinking Questions.

“Why is this important and what, exactly does it mean?” is one of the most powerful questions you can ask about your mobile testing goal.

For example, which devices should you test against? The answer depends on which devices your current customers or prospects are most likely to be using. You know that mobile apps behave differently on different devices and operating systems, right? You won’t be able to afford to test against every device out there – you’re going to have to make some assumptions and then test their validity.

The more you can refine your mobile testing goal by clearly understanding what is directly relevant (need to have) and which things are “nice to have”, the more effective your mobile testing strategy will be.

Wait Just a Second – Why Bother with a Goal?

You don’t really have to have a goal to build a mobile testing strategy.

Your strategy could be “just test it”,or “get folks to bang on it a bunch”, or even “meh, just test it enough to make sure it doesn’t crash”.

Those are all valid mobile testing strategies and they have even been successful strategies – a couple times.

Here’s the problem that happens when you don’t have a goal.

You Don’t Know When You Are Done.

So, unless you want to test forever or don’t really care about your app or your customers, you’ll probably benefit from the 10-15 minutes it will take you to build a good goal for your mobile app.

A List of Things to Decide for Your Mobile Testing Strategy

Since you’ve made it this far, you probably realize that your questions are far more important than your answers!

Here is a list of questions that you can use to help you narrow your focus and build an effective mobile testing strategy:

  1. Which devices are important to my target audience?
  2. Do I have to have the real device or can I use an emulator?
  3. Can I use a tool for testing and do I have the experience to use it properly? Which tool will meet my needs and how can I tell that?
  4. What network connectivity is required for my app? Wifi? Cellular Data?
  5. Which parts should be automated testing and which should be manual testing? What’s the proper balance and how will I know?
  6. Will I need performance testing for my mobile app? Why would I need performance testing and which parts of my mobile app are subject to network congestion?
  7. What are the security requirements for my mobile app and how do they need to be tested? What is the probability and impact of a security breach of my mobile application?
  8. What sort of UX testing do I need and who is best qualified to give UX feedback on my mobile app?

Whew! Those are not easy questions!

And yet, you do need to know those answers if you want to avoid a “just get it tested” type of strategy.

So, is there an easy way to get started and build a great mobile testing strategy?

Additional Considerations when Selecting a Mobile Application Testing Company

When customers are selecting a mobile application testing company to work with, there are several key factors they should evaluate; failing to do due diligence on your mobile application testing company of your choice can lead to bad quality control and an overall negative experience with your app. A good mobile testing company will improve the functionality, usability and security of your app in general.

First, you should ensure your mobile testing provider has trained mobile testers or test engineers. This should be a given, but dealing with companies that don’t have trained mobile testers/engineers can be very expensive in the short and long-term health of your app. A company with competent mobile testers will have expertise in functional and usability testing. Additionally, companies with competent mobile testers will ensure that test optimization is done based on the current mobile device market.

Also, the company you work with should have the correct resources for your app’s testing. This can include a lab for device, emulator and simulator based testing. It’s important to talk with your provider about the specific tests you’re looking for and ask about any other testing they provide and how that can benefit your application. You should mention the platforms and operating systems that your app primarily runs on to ensure all testing is done across all verticals. The most important resource that a provider should have is test engineers, QA managers and other staff. For many businesses, it doesn’t make financial sense to hire more team members to their QA team so working with a provider that has all of the expertise you need is vital. With an expert QA team at your disposal, you can get daily status reports on your mobile testing and solve any issues that arise in the meantime.

Lastly, it’s key to work with a provider that understands your business goals and tailors their service to fit your needs. This seems intuitive, but when you get on a phone call with a provider, they should take some time to figure out your business model, specific application needs, etc. This will allow your provider to form the right QA coverage for your product and you can have a rough outline of how their services can fit your needs

A Simple Strategy is Simple!

One of the most common mistakes about strategies is that they are complex. Sure, your strategy can be complex but it doesn’t have to be complex. It depends mostly on the complexity and difficulty of your goal.

We prefer Simple Strategies and we can help you have one too.

For example, if your goal is to build a full-featured trading app for mobile devices, your strategy might be quite complex. But, if your goal is to launch an app that trades only 1 stock, your strategy might be simpler.

Our rule of thumb is that a strategy can be easily explained with three sentences or less. Sort of like this:

  1. we’re going to achieve THIS GOAL
  2. with these CONSTRAINTS
  3. by doing A and B at the same time and then C next and D last.

It should be clear and understandable to everyone involved in your project that doing A and B at the same time, followed by C then D, will achieve the GOAL within the CONSTRAINTS.

If it is not clear and understandable, then dig a little deeper, find the unknown or un-agreed upon aspect, and build a tighter, clearer strategy for your mobile testing project.

If you skimp on your strategy, you can expect to have problems with your plan.

If you invest wisely, and build a powerful but simple strategy, your plan will be obvious and tracking your mobile testing project will be very easy.

And, one more note about building a simple strategy – expect to rebuild it at least once.

Here’s why – the most unknown and uncontrollable part of your strategy will be the Environment. The Environment is everything around you that you can’t control. These can be as simple as a misunderstood requirement or an emergency family leave for a critical team member. They can also be as complex as a major business setback or the cancellation of a critical software license. These are all aspects of the “Environment” you’re working in and can have a big impact on your strategy. When the Environment changes, re-evaluate your strategy and rebuild it if the changes impact your mobile testing strategy .

Start with a Free Scoping Session with Me

I know, that sounds generous and it is.

And, we can do that for and with you in one of our standard 60-90 minute Scoping Sessions.

Most of our clients and prospects don’t have a quality goal prior to meeting with us. Only one or two have ever had a quality strategy when they first came to us.

So, we’ve become very adept at helping our clients and prospects define their quality goal and build an effective testing strategy.

We do this by gently asking you a lot of questions. Sure, we want to know the details about your mobile app and how you want it tested. We can and will do that for you.

And, we want to make sure we provide you with everything you need to achieve your goal. So, the first thing we do for you is clearly define your goal and then propose a couple of different strategies for you to consider. We’ll always recommend at least one of the strategies but we want you to know that there are different ways to achieve your goals and that you have the last word on how we work for you to achieve your goal.

If you’re curious about how a mobile testing strategy can improve your mobile app testing, achieve your goals and build your business stronger, please reach out and let’s talk about how we might help you. Send the form on this page to schedule a call, or give me a shout now at 888.254.9709 .

Mobile App Performance Testing

Do I Need Mobile App Performance Testing?

/0 Comments/in /by

In This Post

Mobile app performance testing is the process of testing and reporting the time it takes your application to process a request or take an action – especially when large groups of users perform the same or similar action simultaneously.

Performance testing answers the question, “When and where does the performance and speed of my application degrade the user’s experience?” It also answers the question, “How many users can I have before my application crashes?”

Performing “At Scale”

“At Scale” is the current lingo for “many, many, many users and transactions”. Performing “At Scale” is a big concern when building software and especially mobile apps. Mobile app performance testing helps you understand the scale your mobile app can withstand before degrading or crashing. Most software development teams invest significant time and money engineering their mobile apps to perform at scale from the very beginning. This is generally a wise investment but only if you are willing to apply mobile app performance testing throughout the software development process. Otherwise, the software development team will not have the feedback they need during the software development process to measure and correct the scaling assumptions they had to make while building your software.

Performing “Good Enough”

If “At Scale” is not a critical business and technology driver for your mobile app, you still might need to be concerned about performance and mobile app performance testing.

Here’s why:

  • All mobile applications have performance limits
  • When mobile application performance degrades and slows down, users abandon mobile apps.

You’ve invested serious time and money building your mobile application. Don’t you want to know how many people you can satisfy with your mobile application before they become dissatisfied?

There’s a big difference between engineering and performance “At Scale” and “Good Enough”.

Engineering and mobile app performance testing “At Scale” is an ongoing endeavor to insure that your mobile application will always be available to as many users as possible. This is an expensive process but worth it when your goal is performing “At Scale”.

Mobile app performance testing  for “Good Enough” is an end-of-cycle process that determines “How many users can I handle before I need to turn on more servers?” This is a much simpler and less expensive process that gives you a finite number and an alert for when you need to add more servers in your AWS or Rackspace cloud.

If you aren’t required to engineer, build and deliver “At Scale”, then “Good Enough” may work best for you. A short engagement to determine “Your Maximum Number” will help you recognize and avoid a disaster where your mobile application performance degrades and you lose customers. You’ll use “Your Maximum Number” to alert you to add more server capacity before your known performance threshold and prior to any perceived slow downs by your customers.

What Does Mobile App Performance Testing Look Like?

Mobile app performance testing is a simple process but it’s not easy.  Here’s one way of looking at the process:

  1. Build your mobile app features
  2. Add “Instruments” at locations within your mobile app that will measure throughput
  3. Build or find a “Pump” to push specific user scenarios through your mobile app
  4. “Load” the “Pump” with one or more test cases or user scenarios
  5. Tell the “Pump” the volume and timing of test cases to push through your mobile app
  6. Record the readings from the “Instruments” as the “Pump” pushes a measured volume of test cases through your mobile app.
  7. Analyze the readings from the “Instruments” to determine if
    • throughput performance was acceptable – add more volume and retest
    • throughput performance was not acceptable – determine bottlenecks to discuss with your mobile app developers to fix for greater throughput.

Simple But Not Easy

Yeah, it’s simple but it’s not easy. Generally speaking, finding the “Pump”, placing the “Instruments” and building the “Load” of test cases is straightforward. You’ll wind up with a big pile of data from the instruments and you’ll have some early anecdotal evidence of how your mobile application performs under a load.

But, it’s seldom actually that simple because mobile applications almost never just fail due to a high load. It’s the nuances of what happens under one set of load circumstances that look just fine and another set of load circumstances that crashes your mobile app. This is when a trained and experienced Performance Testing Engineer can really help you and your software development team quickly find, test and fix the performance bottlenecks in your mobile application.

Mobile app performance testing can be a critical component of testing and launching your mobile app. Not every mobile app needs performance testing, but when you expect a large volume of users or heavy usage, mobile app performance testing by experts can save you from a major crash and downtime where you can lose customers and prospects.

Look Before You Leap

You want good testing results, of course. Or put another way, you want good testing value. That’s why it’s crucial to define the mobile testing strategy first, which includes performance testing, before thinking about a test plan.

Want to discuss your situation? Call me 888-254-9709. Or, click the blue and green button on this page to request a call back.

Mobile Application Testing Services Choices

How Much Testing Does Your Mobile Application Need?

/in /by

Mobile applications are different than web or installed applications and deserve some careful thought to make the best choices about what and how much mobile application testing services are really needed.

There is a long list to consider and surprisingly, you probably don’t need all of those mobile app testing services to achieve your goal.

Mobile application testing services can include:

  • Mobile application functional testing
  • Mobile application regression testing
  • Mobile application cross device testing
  • Mobile application test automation
  • Mobile application security and penetration testing
  • Mobile application performance testing
  • Mobile application usability testing

Goals for Mobile Application Testing Services

Here are some questions you can consider to help you determine if one of the types of mobile application testing services I listed will help you meet your needs and achieve your goal.

First, what is your goal for your mobile application?

  • How many downloads or users do you want to use it?
  • What value will your users receive from using your mobile application?
  • How will you know they have received that value?

Now that you have a basic goal for your mobile application, see if the following questions help you chose the best and most appropriate mobile application testing services.

Mobile Application Functional Testing

Does you mobile application contain complex or simple functionality?

Simple functionality with only a few inputs/outputs/screens usually doesn’t need anything other than some basic validation that the inputs/outputs and screens work and flow properly. There are few inputs/outputs/screens so the number of combinations to test will be limited and can be accomplished relatively quickly.

Complex functionality where you have more than just a few inputs, outputs and screens probably needs (and deserves) a more robust testing strategy and plan. In general, you can’t test everything (it takes too long and costs too much) so selecting a smart test strategy will be very important to make sure that all of the “Important Stuff” is tested thoroughly and frequently. Almost any qualified Test Engineer or QA Manager will have a small library of Test Strategies and can help you select the one or two that most closely match your needs and goals.

Mobile Application Regression Testing

Regression Testing helps you make sure that you didn’t break anything that was previously working well. The main goal of regression testing is to insure that you don’t disappoint or anger your mobile application users instead of delighting them with new features and functions.

Mobile applications tend to be updated frequently. One of the benefits of a mobile application is that they tend to be smaller and simpler than other types of applications and can be updated very quickly and easily. This is a tremendous benefit because it allows you to quickly add or change features based on real-time user feedback.

But, being very quick to make changes to your mobile application may also hurt you if you don’t take just a little extra time to complete an effective regression test.

Regression Testing Can Be Challenging

  1. Familiarity breeds contempt. When you’ve seen and tested your mobile applications features and functions every day for the past six months, it’s easy to let your assumptions and urgency take control. This can result in skipping or overlooking some functionality in your haste to release your new mobile application. Using an experienced and trusted third party for your regression testing will insure that nothing is skipped (or assumed to be OK) during your regression testing. “Once over lightly and quickly” is not a good regression testing strategy!
  2. Lack of test documentation. Mobile applications tend to be smaller than web applications and good test documentation is usually the first casualty. And no, your mobile application doesn’t necessarily need a full set of ISO compliant test documentation. But, a decent list of test scenarios and use cases can help you complete your regression testing quicker and with a much higher (and measurable) level of quality and customer delight. An experienced third party software test engineer can quickly and easily help you build and maintain these test documents as a natural part of helping you regression test your mobile application.

Regression testing is simple but not always easy if you aren’t an experienced software test engineer. But, getting help from a professional, at least the first couple times, can go a long way to helping you release your mobile application faster, better and cheaper.

Mobile Application Cross Device Testing

Your mobile application might be used on a new iPhone or an Android phone several years old. Almost all mobile devices behave slightly differently and the difference in behaviors between iOS and Android can be significant. If you fail to test your mobile application on a variety of mobile devices, you are inviting a catastrophe.

Cross device testing helps your mobile device safeguard against the following.

Customer loss: Cross testing is vital to ensuring quality assurance across all platforms that your application is on. If everyone was using the same device with default graphics, this wouldn’t be an issue. Users today use a variety of devices across several platforms; cross testing across platforms ensures users from all ends have the same experience. If the user interface on one platform performs poorly, customers will likely leave the app quickly.

Issues that arise from upgrades: Mobile applications are updated very frequently and these updates will eventually cause several bugs. Cross device testing will help you identify bugs and how to deal with them. Failure to recognize this might cause significant bugs that can cause your application to be unusable until it is resolved. This can also lead to compatibility issues after design changes as well.

SEO and additional rankings from Google: For mobile applications, ranking on search engines is crucial to gain traction to your site and mobile app. Since 2015, Google has taken a mobile-first indexing approach. This means that the algorithm for Google search results takes cross device compatibility into account and ranks accordingly. If a competitor to your app is compatible with all platforms and devices, they will significantly outrank you.

There are two basic methods available for cross device testing. You should consider the pros and cons of both before specifying what mobile application testing services to obtain.

Real Mobile Devices and Operating Systems

Real devices provide the best mobile application testing results since you are testing your application on the actual device. But, Real Mobile Devices are expensive to acquire and maintain. Real Mobile Devices are also slightly more time consuming to test on as they require physical handling. Finding a sufficient variety and availability of Real Mobile Devices is a big challenge as new devices and mobile operating systems are released and updated frequently – and this makes the process of building and maintaining a library of mobile devices very expensive.

Mobile Device Emulators

Emulators are usually on-line services where you can load your mobile application on a variety of specific virtual or emulated mobile devices with specific operating systems and versions. Mobile Device Emulators are very handy but have some distinct challenges. Mobile Device Emulators work well for basic functionality but are seldom useful if hardware functionality testing (GPS, Cellular Vs Wifi Networks or Device rotation) is needed to test your mobile application.

Also, mobile device emulators can’t emulate the performance of an app when it receives incoming calls, text messages or other notifications. This is a crucial issue because everyone with mobile devices gets numerous notifications every hour. If an app has to crash each time a notification occurs, this would make it unusable. Additionally, mobile device emulators can’t simulate other aspects like battery issues, touchscreen issues and more. These are some of the most fundamental aspects of using a mobile device.

But, We Think There is a Better Way

The TESTCo Blended XDT (Cross Device Testing) – at TESTCo, we take a blended approach to using Real Mobile Devices and Mobile Device Emulators. We will help you analyze your expected customer device usage and the current market share of each mobile device and operating system and produce a matrix that compares your testing needs against Real and Emulated devices. We use this to build a testing strategy that optimizes for primarily Real Mobile Device Testing with a portion of Emulator Testing so that we can give you the highest confidence that your mobile application will work flawlessly on the devices that your customers use.

The TESTCo Blended approach takes advantage of the benefits of both real mobile devices and emulators while covering the blind spots of each approach. This gives your mobile application testing a comprehensive approach that would simulate what would happen to your application in the real world. This ensures that your mobile application will work seamlessly during its first encounter with users doing normal functions on their phone like using it with low battery, taking calls while using your app and more.

Mobile Application Test Automation

Using test automation for your mobile application can allow your software development team to build and release your software quicker. But, is development speed really a problem and do you want to invest to solve that problem?

Here’s why I ask. Test automation is the process of writing computer code (for the test automation tool to use) to test your computer code (your mobile application). So, you’ll be building another computer program (to test your mobile application) that does test your software faster but is more complex and expensive to build and maintain. Go into this one with your eyes wide open for the ongoing consequences and not just the initial expected benefit of faster testing.

Another issue with “going faster” when building and testing your mobile application is the approval and update process – which is entirely out of your hands! Sure, you can build and test your mobile application faster but you still have the issue of bottlenecks getting your mobile app approved for Apple or Google store and then, even after it has been re-approved and loaded in the Store, your chances that your users will actually download and install the update are small.

So, instead of “going faster”, you might find that “building smarter” provides you with more benefits and fewer costs. Nevertheless, building and running test automation can be valuable for your mobile application in certain situations.

Mobile Application Security and Penetration Testing

Your mobile application probably has some personal and/or confidential information on it. Hackers are penetrating, stealing and selling confidential data from mobile applications and their servers on a daily basis.  Can you afford for your mobile application to be hacked and your users mad at you?

Thankfully, most mobile applications have a small number of security and penetration points so testing them is common practice.

The security of your mobile application’s login screen probably isn’t an issue. In fact, most mobile applications are naturally quite secure in their user interface as long as the login function is well built – and most are.

The real challenges with mobile application and security lie in the way your mobile application communicates with the world around it and it’s server. Mobile applications use API calls to send and receive information to and from other users and the server in most cases. The robustness, error checking and reliability of these API calls will determine how “safe and secure” your mobile application will be for your users. Mobile application security and penetration testing evaluates your mobile applications login screen and internal functions as well as your API calls to ensure that your customers’ data is safe and secure.

Penetration testing can be done against different types of code and systems in your mobile application like servers and APIs. Depending on your goals for the pen tests, there are several phases within each pen test to ensure that the security of your application is covered from all angles. After a pen test is done, a formal report will be given to you that covers all the vulnerabilities that were found and how hackers can use them to gain access to your application’s data. In contrast to many systems and processes of your mobile application, penetration testing is largely a manual process. There are testers that overlook the entire process to make sure that your mobile application cannot be breached.

Mobile Application Performance Testing

Your mobile application will need to respond and perform quickly for your customers and prospects to feel confident using your app. Poor or questionable performance can (and does) lead to poor reviews, fewer stars and rapidly decreasing downloads and usage. This probably is not your goal so performance testing may be a risk you want to consider selecting for your mobile application testing services.

There are three critical areas to consider when performance testing you mobile app.

Device Performance  – Your mobile application needs to do 2 things simultaneously – respond quickly when in use and play well among the other mobile apps on the device. Testing the first is usually straight forward. Testing the second will require a smart strategy to get accurate results quickly without wasting a lot of time with invalid test setups.

Server/API Performance – Your mobile application will need to communicate with your server to send and retrieve information. In most cases, the server application is separate from your mobile app and interacts with your mobile app through an Application Programming Interface (API). This server app will tend to behave like most server-based applications and is subject to the same performance testing with one exception. The APIs need special attention to a wide variety of performance loading criteria to make sure the mobile apps that depend on them are not needlessly bogged down. Make sure to remember to test the server under load as well – not just the server application!

Network Performance – There are almost too many mobile networks to even list anymore. Mobile devices communicate either via WiFi or the carrier’s mobile data network. In all cases, even Wifi, the network speed and capacity are going to vary wildly. 2G and 3G networks still exist in many places in the world and the continental United States. Many WiFi hotspots are still using the older 802.11(a) and (b) technology. Network performance testing is tricky and it is critical to understand the user’s environment and goals before building a network performance testing plan.

Lastly, all performance testing requires detailed data analysis to clearly identify and truly understand the performance bottlenecks. Performance testing typically produces massive amounts of data that needs to be analyzed to be understood. Performance Test Engineers specialize in this data testing and analysis and have spent many years learning and building their knowledge.

Mobile application performance testing can be tricky to plan and execute so that you get the full value of the work. There is a lot of performance testing that can be done but typically only a few of the tests are truly valuable to any certain situation. The key is to be perfectly clear on the problems you want to solve and the risks you want to reduce. This approach allows your Test Engineers to perform the exact tests you need without wasting time and money on unneeded mobile app performance tests.

To Be Continued

In the coming weeks I will shed light on the five other types of mobile application testing so you can make smarter testing decisions. In the meantime, check out this post if want tips on how to select a mobile application testing company.

If you can’t wait for me to finish this post (hey, I can’t write all day, I have a business to run), call me (888-254-9709), or request a call back using the green button on this page. I’m happy to answer your questions.

Mobile Application Testing Strategy

Why You Need a Mobile Application Testing Strategy

/in /by

In This Post

App testing looks easy but you might want to think again – and I’ll help you think through a mobile application testing strategy.

Simple, but not Easy

You can very easily skip or ignore some very important risks just because you didn’t even know to think about them! Mobile application development moves very fast, is usually highly agile and covers much more than just the mobile app itself. Most mobile application testing projects spend too much time on functional testing and not enough time finding and reducing real technology and business risks.

Starting to test your mobile app without a Testing Strategy can result in a “well tested” mobile app that frustrates customers, scares off prospects and lights up the support phone and chat lines.

All because you didn’t know what you didn’t know!

And that’s probably not what you wanted or expected.

A simple mobile testing strategy will help you find the important risks, plan your testing work to focus on the important areas and reduce the time and money you spend launching your mobile application

What You Don’t Know CAN Hurt Your Project!

There are six  important subject areas in that you need to consider in your strategy for testing your mobile application.

  1. Devices – What is your list of most important devices? What devices are you willing to exclude or not test? Do you have access to those important devices?
  2. Networks – What networks will your mobile application use? Are they cell networks or wifi networks? What happens when your mobile app switches networks?
  3. Servers – How much of your mobile application functionality depends on accurate information from a server application? Most mobile apps rely heavily on back-end servers to do the heavy lifting work while the mobile app itself mostly accepts inputs and display outputs. What happens in between your server and mobile app?
  4. Performance – What are your minimum performance times for functions in your mobile application? What are the upper limits of your server? How many simultaneous users do you expect? What performance tools are provided by your server hosting company?
  5. Security – How secure does your mobile application need to be to protect your user’s information privacy? Is 2FA important to your users? How secure does your server need to be?
  6. UX (User Experience) – Will your users find your mobile application enjoyable and easy to use or frustrating and confusing to use? What UX data will you collect to help you understand this? If a user does leave, how will you recover them?

If you know the answers to these questions, you’ll be able to clarify your goals and build a mobile testing strategy to achieve them for your unique mobile application.

Curious yet about TESTCo’s mobile app testing services? Here’s more information on our capabilities.

No Mobile Application Testing Strategy? Here Comes the Pain.

If you don’t answer those questions, you can expect to see (and feel) some of these pains:

  1. Your mobile works fine on the most recent iPhone but fails with previous versions of iOS and some Android devices. And, only a small portion of your audience has the most recent iPhone.
  2. Your mobile application works great on Wifi but fails with errors on a cellular network if a user is talking on the phone and using your app.
  3. Your mobile application fails and crashes when the server is slow returning complex query results.
  4. Your mobile application works fine until there are over 900 simultaneous users – and then it displays the wrong query results.
  5. Your mobile application is easily penetrated and accounts are hijacked.
  6. Your mobile application UI works great for the engineers who built it but ordinary users struggle to understand how to use it.

Build Your Own Simple Strategy – or Let Us Help You!

The best solution is to build a simple mobile application testing strategy unique to your project. It’s easy if you follow this guide I recently published.  At TESTCo, we build and evaluate several test strategies for every software and mobile application testing project. If you don’t want to build your’s, we can build it for you!

Schedule a call with me using the green button on this page, or call me at 888-254-9709

Mobile App Testing

The Mobile App Testing Lessons Learned in 2018

/0 Comments/in /by

In This Post

The market for mobile applications is continually shifting. Failure to get it right, the first time, and keep it right, results in pretty severe consequences. In 2018, TESTCo ran tens of thousands of tests on our clients’ mobile applications. Here’s what we learned. Plus, I’ve included a link to our Mobile App Testing Checklist.

Lesson #1 – Quality Matters in Reviews

The quality of your mobile app will directly influence your app’s ratings and reviews. Users definitely use the ratings and reviews to determine if they should download and use your app. Poor ratings and reviews, especially regarding quality, will cause users to avoid your mobile app.

Improving ratings and reviews, especially pointing out improvements in your mobile app, will cause users to give your app a try. Ratings matter and the quality (of lack thereof) have a big impact on the quality and quantity of your reviews.

Lesson #2 – Choose Your Test Devices and Browsers Carefully

There is no single reference page for determining which devices, operating systems and browsers have the most market penetration. Nobody really knows how many people are actually using any particular flavor of mobile device or operating system. You can’t test every one so how will you know which to test and which to avoid? If you test the wrong group of devices and browsers, you’ll have defects.

Download Mobile App Testing Checklist

Lesson #3 – Don’t Stop! Keep Going! But Not Too Much!

The mobile app market is moving quickly and still gaining steam. Without a real and almost daily need to use your mobile app, users will quickly forget about your value proposition and move on to another offering or app.

If you refresh you mobile app, your users will fall in love with you all over again. Except! Not too often. Too many upgrades and your mobile app users will avoid your updates and miss your new value propositions/features or become confused and delete your app because it appears buggy.

The Preventable Reason Mobile Apps Fail

Mobile apps are almost a strict business requirement – like a web page for a business but harder and more expensive to maintain. They also allow you and your business value proposition to sit, literally, in the pocket of your customers! Get it right and your cash register will ring. Miss a few steps or deliver poor quality and they’ll slide right by and go to your competitor.

Give your innovation a chance to succeed. The #1 reason that mobile apps fail is poor testing. We can help you do it right the first time.

Automated Mobile App Testing

When Is Automated Mobile App Testing The Right Solution?

/in /by

In This Post

Automated mobile app testing seems like a silver bullet to the problem of testing your mobile app quickly and inexpensively. After all, computers can run tests faster than humans and don’t need breaks or time off. And, seldom does a day go by that you don’t hear a story about a business using process automation to improve their speed to market and lower their human costs. Seems like a good idea, right?

Well, one of my business mentors taught me that all my biggest problems started as a good idea that I failed to explore in detail before starting on it.

Here are three things you should consider before starting your automated mobile app testing.

Issues To Consider About Automated Mobile App Testing

  1. Automated testing is more expensive than manual testing but the cycles times should be shorter.
  2. Automated tests have to be constantly maintained and updated just like your mobile application.
  3. Automated tests with false positives are worse than no testing at all.

Speed Versus The Rise of False Positives

One of the primary benefits of automated mobile app testing is the speed at which the testing can be completed. Faster testing cycles means faster mobile app releases which allows you to seize more opportunities. The higher up-front costs for automated mobile app testing should be offset by faster release cycles. But, is your mobile app development team really ready to go faster? How much faster and what might happen if it’s too fast?

A distinct drawback to automated mobile app testing is the constant need to update your automated test scripts to match the new and changed functionality in your mobile app. An automated mobile app testing system can only test when it has been told to test and nothing more. If you change your mobile app, then you need to change your automated mobile app tests – every time.

And a common failure in automated mobile app testing is the rise of false positives over time. A false positive is when your automated mobile app testing systems reports a defect but it isn’t really a defect. Your mobile app is correct and your automated mobile app testing system is wrong. It is probably out of date and trying to test the last version of your mobile app and not the current version where functionality has changed. This results in false positives that have to be categorized, prioritized, verified and then resolved. If your automated mobile app testing system is reporting false positives, you will waste and enormous amount of time and money chasing them down.

The ideal situation for automating your mobile app testing is when you have a relatively stable mobile application and you and your development team are committed to increasing your process automation. The best reason to automate your mobile app testing is to improve your software release speeds and reduce human error due to repetition in your testing cycle.

Automating the testing of a relatively stable mobile app produces far better results than trying to automate a young or new mobile app. There is a minimum investment required to build a test automation harness that will then be filled with automated tests. A young mobile app presents a very difficult target to automate due to the highly dynamic and changeable nature of the app. A young mobile app, by definition, is subject to numerous changes as the team works to try a variety of different approaches to please their customers. Automated testing for a stable app is easier to build, less costly to maintain and more accurate in it’s results.

The Impact You Can Expect

Make no mistake, automated mobile app testing will have an impact on your development process. You’re probably using a small collection of dev automation tools to help you manage your codebase and produce a releasable version. Adding automated mobile app testing into your automation stack will require someone on your team to manage and integrate it into your overall system. You and your team will need to be ready for this.

Lastly, automated mobile app testing is more expensive than manual testing but it is faster and more accurate. Since that’s the case, the only really compelling reason (and it is compelling in a *lot* of cases) is to increase the speed and reduce the time needed to release new features into your mobile app. If getting “faster” isn’t a distinct market advantage, it may make more sense to look at other alternatives to automated mobile app testing.

Confused Yet?

Automated testing, especially for mobile applications, is an important decision and worth evaluating (and re-evaluating) on a regular basis. It can be an expensive investment and you’ll want to make sure you’re getting the biggest bang for your buck.

The best place to start is a conversation about what might be possible for you, the goal you’re working to achieve, the strategies available to you to achieve your goal, and the costs and benefits of each strategy.

I help managers straighten out their mobile testing strategies every day. Sometime I even get paid for it.  If you’d like to talk about your options, I’d  like to listen. Call me at 888-254-9709, or click the green button on this page to request a call back.

Mobile Application Regression Testing

Why Regression Testing Must Be Included in Mobile Application Testing

/in , /by

Are your mobile apps prone to breaking after implementing a new feature or update? If so I encourage you to make mobile application regression testing a regular part of your test plan?

You see, after an update or new feature has been added there’s a really good chance that you have some new defects in your older, previously working code. Why might that happen? There are several reasons that your previously “good” mobile application now has new defects after your last update.

Why Defects Appear in “Good” Mobile Applications

  1. Changes in the mobile operating system – it is not uncommon for changes and improvements in a mobile operating system to have a negative impact on your mobile application. This isn’t intentional nor does it happen every time. Testing on a variety of mobile operating systems and devices can help you find these regression defects quickly. Knowing which operating systems and devices to test is where professional mobile application regression testing can help you save time and money.
  2. Changes in code module dependencies – most mobile applications are built by assembling a set of basic software components and then customizing how they interact with each other and your customer. These basic software components are almost always being changed and improved. When those new changes and improvements are included in your new mobile application version, errors are likely to occur. Writing test plans to specifically test mobile software component interaction will help you find these types of regression defects. Professional software test engineers know and use effective test strategies and plans to help you identify and test these “risky” areas.
  3. Human error in development or testing – humans make mistakes – sometimes knowingly and sometimes without knowing. Most regression defects caused by human error are not intentional – in fact, just the opposite! Human errors typically occur when an improvement in one area of the code has a negative impact on some other area of the code that wasn’t foreseen by your software developer. For example, improving the user experience for logging in may have an unexpected impact on how a password reset request works that cause it to fail. QA managers, as an integral part of a mobile application regression testing team, can help you better understand how your requirements/user stories can show you clues to find these types of regression defects.

Smart Mobile Application Regression Testing Begins with a Smart Test Plan

Regression testing doesn’t fit easily into an Agile software development process. And yet, going without regression testing for your mobile application can lead to loss of prospects, customers, and revenue. Working with the right outsourced mobile app testing company certainly goes a long ways to removing the risk associated with rolling out mobile app enhancements.

Go ahead, call me. 888-254-9709. Even if you just want to pick my brain, I want to hear from you. Or, if you prefer, click the blue button on this page to schedule a call back.

 

Mobile Application Testing Company Selection

How to Select a Mobile Application Testing Company

/in /by

Mobile application testing is arguably more important than regular software testing.

Here’s why. Most mobile applications are free and most software is paid for. Additionally, most mobile applications are designed to attract new customers and retain your current customers by making it easier to buy from or work with your company. Most software is used within a working environment. See the difference? Mobile applications primarily attract and retain customers where most software is used to accomplish a task.

Now, defects or bugs in ANY software are a problem but defects and bugs in mobile applications have a much higher cost than most defects in regular software.

Why?

Because people seeking help or a new source to purchase from have ZERO TOLERANCE for defects that make it hard to begin or continue doing business with your company. If they find a defect in software they’ve purchased from you, you’ll get a help desk call. If they can’t get your mobile application to work, what do they do? Think about it for a second – what happened the last time you encountered a defect in a mobile application. What did you do? Yeah, you probably closed the mobile application and went somewhere else. I do.

The cost of a defect in a mobile application is higher than the cost of a defect in most software. The cost is a lost customer or new prospect

Mobile application testing is little different from software testing in a couple of  other key areas, as well.

“Walled Garden” Software Distribution Challenges

Releasing your mobile software requires “approval” from a third party prior to listing on Apple Store or Google Play. How will this requirement impact your release dates and software testing schedule? What do you need to know to avoid Apple or Google declining your mobile app?

Inputs & Navigation Vs Display & Calculation

The intense focus on Inputs and Navigation in mobile apps requires different software testing strategies to avoid wasted time and effort. How will you know you’ve tested the most vulnerable areas in your mobile app?

Devices, Browsers, Networks and Emulators

So many and growing every day! Your mobile software will operate slightly differently on various device, browser and network combinations. Which combinations are required and which are optional? Which combinations need testing on a real device or network and which can be tested on an emulator? Making the wrong choice can cost you prospects and customers!

Test Engineers & QA Managers Vs Testers

Almost anyone that has used a mobile device “can” test your mobile app. But should they? Test Engineers coupled with a QA Manager can give you full QA coverage and Risk analysis. Testers can usually do what you tell them to.

Not every software testing company will know these key areas well enough to meet all your needs as a mobile application testing company.

Where to start?

There are many mobile application testing company choices available to you. What should you consider and where should you start to make your selection?

  1. Clearly define your quality goal and your budget for time and dollars. This will help whoever you select to build an accurate proposal/estimate. There will always be more QA work that is needed but business decisions drive mobile software projects. Time and money are key constraints that limit what can be done.
  2. Mobile application testing experience is critical. Mobile application testing is different than testing regular software and requires different tools, patterns and methods. Likewise, mobile application developers are seldom a good choice for testing your mobile application. Software test engineers are your best choice.
  3. The size and recency of the device and emulator library will dictate how successful a mobile application testing company can be for you. The wide variety of mobile devices and operating systems dictates that you make two wise choices.
    • Which devices and operating systems are most important to you and your customers?
    • Which devices does your mobile application testing company actually have? When a particular device/operating system is not available, an emulator must be used instead. Emulators are useful for some testing but almost always fail to provide the complete testing coverage that most modern mobile applications require

The Cost of a Bad Choice

You must already be convinced that testing your mobile application is important and needed – as evidenced by your visiting this page.

Congrats! Some of your competitors either don’t bother to test their mobile application or do it poorly.

Don’t believe me? Just download one of them and see how long it takes before you can crash it. Not long, right?

Here are the big costs you need to avoid when selecting your mobile application testing company.

  1. Price Vs Cost – in software testing, you really do get what you pay for. Low rates/prices are the teaser and longer projects, poor defects, missed coverage and blown out deadlines are the true cost. Compare their calculated project cost instead of the hourly rates. Then,  get a money-back guarantee to make double sure. Even better, get a fixed price for your project.
  2. Talent Vs Availability – not everyone is a talented Test Engineer. In fact, most “testers” aren’t engineers at all. If your problem can be solved by “just get it tested” then almost anyone will do – and there are plenty of those folks out there. If your goal is “on time with zero defects” then you’ll need talent – Test Engineer talent and QA Manager Talent. And yes, the price for a Test Engineer is higher but see #1 above.

The Real Cost of Working with a Bad Testing Provider

When doing mobile testing, there are several strategies you can employ. You can test the application yourself, have developers test it, have a 3rd party team test it or you can choose to not test it at all. Out of these choices, having an expert 3rd party team is generally the most cost-effective option; this allows you to access expert testers and get your application into the market as soon as possible. While the ROI of a good 3rd party testing provider is great, working with a bad testing provider comes with its challenges.

The costs of working with an incompetent mobile testing provider is multi-layered. First, there’s the obvious cost of hiring the testing provider to test your mobile application. Prices for selecting a provider can range between companies, but it’s important to be cautious about providers that may charge well below industry prices or providers that are located overseas. While the upfront cost may be lower than other providers, the back-end cost of failed testing and wasted time in addition to the upfront cost can be enormous.

Working with a bad testing provider generally accompanies working with incompetent testers. This doesn’t mean that the testers are bad in general, but a regular software tester might be unable to see your application from the code and system level. This is why it’s important to have software test engineers because they have the understanding to develop and implement a test plan that achieves your project goals. 

A good provider is generally very cost-effective. The price of the testing done is reasoned by all the facets of mobile testing that is done for your application. This can include functional testing, performance testing, network testing, cross-device testing, usability testing and more. With each type of testing, there’s a subset of testing that is done to make sure all avenues of mobile testing is done for your application. With good providers, all of these services are provided by competent staff that include testing engineers and QA managers. Having that kind of technical bandwidth and expertise for your application would otherwise require full-time salaried staff and an array of testing platforms.

The Keys to Successful Mobile Application Testing

The three key areas where mobile app testing differs from software testing are:

  1. Faster and Simpler – mobile application projects are usually faster to start and complete since they are smaller and more focused. Applications on mobile devices are also usually simpler in functionality than software applications due to limited screen real estate and uncertain connectivity.
  2. Experience and Completion – mobile apps typically focus more heavily on the user experience than most software applications. Also, clear evidence that the transactions made in your mobile application have been completed are more important due to the tendency of mobile applications to lose network connectivity.
  3. All Those Different Devices – software applications and (most) web applications run in a web browser on a desktop operating system. There are a finite number of operating systems and web browsers that need to be tested against a software or web browser application. There is an almost infinite number of mobile operating systems/browser combinations – definitely more than you could possibly test.

With these three big differences in mind, can you see how selecting the right mobile application testing company might be important in the success of your mobile application project?

Questions for Your Next Mobile Application Testing Company

Here are some great questions to ask when you’re shopping around for a company to test your mobile application.

  1. With our very fast sprints, how will you design your testing strategy to help us increase our speed?
  2. Since we have very simple functionality, how much and what type of testing do you propose to make sure we achieve our goal of Zero Defects?
  3. What strategy do you propose to test the usability of our mobile application?
  4. How will you test and validate transaction completion and notification? How will you test graceful failures?
  5. What is your strategy to help us figure out which devices to test on and which devices that an emulator should be OK?
  6. Is hard device testing necessary or will a tool or emulator be sufficient? Which tools/emulators do you recommend and why?
  7. Where does test automation fit into your strategy and why do you recommend it?

Mobile application testing is similar to but different enough from traditional software testing that a slightly different approach to selecting the best mobile application testing company can mean the difference between your success and failure.

Want to talk with someone at TESTCo about how we’d test your mobile app? Most clients, at first, aren’t even sure how much testing will be necessary. We will give very practical advice in the first call. Call and talk to me personally. 888.254.9709 Or, click the green button on this page to schedule a call back.