Automated Regression Testing for Software | TESTCo

Automated Regression Testing Dilemma

Nobody likes running long software regression testing cycles. It’s long, hard and boring.

And yet, nobody likes to release bugs either. NPS (Net Promoter Score) drop like a rock and the support phones light up like little annoying Christmas ornaments.

So, just grin and bear it or take the plunge into software test automation?

It depends on where you stand on these two dilemmas.

1. Business Speed Dilemma

This dilemma is about your cost and speed of producing software versus capturing more customers and larger market share before your competitors.

  • Adding customers, solving real problems and growing your market share requires more features in your software.
  • More software features require more software testing.
  • More software testing takes more time, people and delays releasing your new features.

So, faster software production equals more customers as long as you meet their minimum quality requirements. Otherwise, they dump your software or mobile app like yesterday’s leftover tacos.

2. People Versus Machines Dilemma

This dilemma considers that people are more adaptable but more expensive than machines that are more consistent but less expensive.

  • People are better at finding defects.
  • People are slower and more expensive than machines but require less “programming”.
  • Defects found late in the process are very expensive to fix.
  • Machines are better are consistently repeating sets of activities.
  • Machines are slower and less expensive than people but require more “programming”.
  • Defects found early in the process are cheaper to fix.

So, how to resolve your dilemma? My friend and business mentor, Keith Cunningham says

One choice is slavery. Two choices is a dilemma. Three choices is freedom.

Hybrid Software Regression Testing That Turns a Dilemma into Freedom

Consider a hybrid approach to software regression testing.

Use people for manual regression for all features in the current and previous version.

New(ish) features are where most of the defects lie in hiding. This leverages the human advantage of finding defects in new features.
This also minimizes the amount of time & money spent on people in the regression testing cycles. You will have the benefit of faster and cheaper software regression testing cycles.

Use Test Automation for regression testing all features released prior to the previous version.

Older features are where most software developers do damage. This leverages the machine advantage of consistently repeating activities. This also minimizes the amount of time between discovering a defect and fixing the defect – which dramatically reduces the cost of fixing the defect.

You and your developers will have the benefit of real time testing so that regression defects (where new features break older existing features) are much cheaper to fix.

The Drawbacks to Hybrid Software Regression Testing

  1. You may need to add Test Engineers to your software testing team. Successful test automation will require written test cases that can be easily automated. Test Engineers know how to write these types of test cases.
  2.  You will have to decide on which approach to use for Test Automation – Record/Play, Scripted or built-in APIs. You’ll need an experienced Test Automation Engineer to help you determine the best path for you and your business.

At TESTCo, we can help with both – call and let’s talk about how to implement or improve your automated regression testing.

agile software testing reportDownload our report that looks at the challenges and pitfalls of creating automated regression tests in an Agile environment.

Software Test Automation

Software Test Automation Case Study

Journyx has customers worldwide and have been improving their products with software test automation for the past seven years. The company offers customers project, time and expense tracking software, and resource management software for forecasting projects, budget status, and employee time and availability.


Before hiring TESTCo to manage software testing and develop best practices, Journyx was in a world of hurt. Early on, they used company staff and even some family members to bang on new software releases, but that didn’t work so well. This was about 15 years ago. “It’s difficult, repetitive work,” said Journyx CEO Curt Finch. “No one really likes it, and it’s not really a core competency.”

The company wasn’t able to hire more people for testing, but they were always under pressure to release new versions of their products. “We couldn’t get a handle on when our next release would show up and what level of quality it would be. We could say we’re going to ship this come hell or high water by May 1st, but we’d be in danger of shipping broken software.”

“Back then, customers hated us and wanted to shoot us,” said Finch.

Journyx’s director of product management, Meredith Zachritz, had to manage a lot of the chaos at the time. “In the early days, our developers were responsible for testing their own code,” she said. “There were no robust peer code reviews. It was very messy. We just about drove ourselves out of business.”

The company didn’t have an official product manager back then, and they didn’t have the budget to hire testing resources in-house. “I was spending more than 50% of my time on testing and related fire drills,” said Zachritz. “It affected customer retention, and we lost confidence in our product’s ability to handle customer needs.” When mobile apps came along, complexity increased, and it was even more difficult to keep up.



The Journyx team eventually decided on outsourced testing, but they struggled with how to do it. Initially they hired contractors in India, but that failed. “We tried a couple of different firms,” said Zachritz. “It wasn’t apparent that they had QA expertise. If they did, the language barriers and time zone limitations were too difficult to overcome, and we couldn’t make it work. Our QA didn’t improve, and they didn’t understand what our goals were. They had no certifications or formal expertise, and no one was managing the entire project.”

Journyx ended up with nothing to show for their efforts – no test scripts, documentation or processes they could hand over to their internal development team. “We got what we paid for,” said Zachritz.

After that fiasco, Journyx hired a dedicated employee to be QA manager. They built up a set of test scripts – nothing automated – and some manual testing methods. “We had to train her,” said Zachritz. “She was only one person. When she was out, she wasn’t doing QA. And when she found a more attractive offer, she left. When you have someone in house, those are always issues.” That employee didn’t offer up any strategic guidance or best practice recommendations either.

Local, Accountable and Low-Cost Solution Helps Drive Revenue

Journyx eventually found TESTCo and solved everything in one fell swoop. They found a local company with the expertise to manage complex software testing and low-cost offshore testing labor. “What I like about TESTCo is that they’re an intermediary to different cultures,” said Finch. “At one point they switched testers from Romania to Ukraine, and we didn’t notice the change.”

“They’re here in Austin, and I can take them to lunch and hold them accountable or beg for forgiveness if we’ve messed something up on our end,” he continued. “We trust them, and the costs are very reasonable for what we get.”

Finch categorized the value of what TESTCo brings in terms of revenue, cost and risk. “TESTCo helps us understand exactly what we’re shipping,” said Finch. “We know customers will be happy with it right away, and our sales people have confidence in the software. That translates to increased sales.”

“The sales team is no longer afraid to demo stuff in presentations,” added Zachritz.

“Our costs are reduced because the testing talent is in another country where it’s less expensive to employ them,” continued Finch. “If I were to replace TESTCo with people here in the U.S., I’d have to spend about $5,000 to $10,000 more per month.”

As Journyx rolls out new product features and new software, they reduce risk by leveraging TESTCo resources. “For one thing, I have reliable, detailed insight into the status of everything,” said Finch. “With TESTCo we’re able to have a known level of quality in a consistent, predictable way.”

Journyx can also examine potential costs and complexities when considering the addition of new software features. “We can ask TESTCo how much it’s going to cost to add something to the testing,” said Finch. “That’s a double-check on the complexity question. It’s like a reverse estimation tool.”

Virtual In-House Team

Zachritz meets with TESTCo for bi-weekly progress calls and has daily email status updates from the TESTCo team. “We coordinate as though TESTCo is an in-house team,” she said. “The TESTCo QA process is run for me. I don’t have to do anything but review status and answer questions. I contact them with adjustments. They come to me with best practices and next steps guidance. All of that is very powerful for me.”

She now spends about 5% of her time on testing chores (as opposed to the 50% she mentioned earlier). “Now I can spend my time on planning and execution, and avoid all that fire fighting,” said Zachritz.

With TESTCo, Journyx can also scale up and scale back resources depending on their release cycles. “We can really manage it closely in terms of what we need at a given time,” said Zachritz. “And it doesn’t hurt us when someone takes a two-week vacation during testing. The TESTCo replacements are seamless.”

Software Test Automation Moving Forward

The TESTCo solution allows Journyx to automate software testing processes, as well. The goal is to reduce regression testing cycle times and get new releases out in a shorter amount of time.

Initially, Journyx didn’t tackle software test automation when starting out with TESTCo, but they eventually decided to move ahead with it in order to further enhance productivity and free up resources. “We tried to implement an automation suite earlier,” said Zachritz, “but coverage was bad, the platform was difficult to maintain, it was open source, and it wasn’t possible to upgrade it.”

TESTCo recommended changing out the platform. The two companies went through a new platform evaluation process, and found a new system that fit Journyx’s budget.

“We ended up automating the most time consuming manual tests and regression test cycles,” said Zachritz. “That speeds up our release cycle and in effect drives sales.”

 The process helps them improve code quality, close the gaps between fixes and generally move faster as an organization. “Our support teams can now work on bug reports and patching custom solutions for customers,” she continued. “They can also work on the videos they create to help people configure their solutions. Software test automation allows us to work on more strategic projects because of the quality of releases.”

“All of it has transformed our business,” said Zachritz. “It’s really incredible how it works.”

Read more about TESTCo’s software test automation services.
software testing companies

Software Testing Companies and Software Testing Tools

Competition between software testing companies is fierce! There’s no better place to see the battle being waged than the front page of Google.  Search for any software testing related search term – especially software testing tools – and you’ll quickly get bombarded with “Me Too!  Look At My Tools!”

Just because a worker’s tool belt is full of tools, it doesn’t make them an expert – not even close.

Software testing tools are a fantastic leverage point for getting more software testing completed in shorter times for less money (faster, better, cheaper).

But, only if they are used for their intended purpose and by someone who has at least a couple successful projects completed.

Otherwise, you just wind up with a situation where “when all you have is a hammer, everything looks like a nail” syndrome.

Don’t Gauge Software Testing Companies by the Tools They Use

  1. Your Tool May Not Work – Most test automation tools favor or REQUIRE a certain technology stack.  If your’s doesn’t match, your tool and project will fail.
  2. Your Tool Handler May Use It Wrong – Just because someone has used a hammer, doesn’t make them a carpenter. Most software testing tools have 2 layers, beginner and advanced. Either way, without proper training and at least a couple successful projects under their belt, your chance of success on YOUR project is low as the tool handler is learning on your dime.
  3. Your Tool Works but Doesn’t Solve the Problem – Test case management tools are great for organizing a bunch of test cases and test plans. But, if it doesn’t also provide the metrics you need to measure quality, it won’t solve your problem.
  4. Your Tools Works AND Solves the Problem but Costs Too Much – This is how most software test automation projects end up – languishing in the background because they person who built it isn’t around anymore to maintain it. You did consider the cost of maintaining your test automation didn’t you?
  5. And Causes Other Problems Later – Test Case Management systems are prone to this issue. They work great at the beginning.  So great that they become integral to your project management and quality assurance procedures. But, because they tend to be tuned to support one development methodology or another (Scrum, Agile, etc.), if or when you change development methodologies, your test case management system presents work flow problems.

So, how can you avoid falling prey to good SEO but poor software testing tool knowledge? Ask more questions!

Here are a few Great Questions to ask your Short List of Software Testing Companies

  1. What evidence and/or experience can you show me that will help me believe that your approach will work for my project?
  2. Who is your tool expert, how did they become an expert, can I speak with them before the sale and will they be assigned to my project?
  3. How can you convince me that this tool and tool expert will effectively solve my problem?
  4. What is the long term cost projection for continuing to use this tool?
  5. Who else has used this tool for a couple years that can explain how well it works over time?

At TESTCo, we believe in honoring promises. We’ve successfully implemented and used software testing tools for almost 20 years. Like you, we expect a software testing tool to solve the problem and keep it solved at a cost that is less than the benefit received from using the tool. If we can’t make the math work, we’ll tell you and offer an alternative. Our tool belt is full and we know when and where to use every one of them successfully!


automate software testing

Automate Software Testing to Deliver Quality Complex Apps

Software and mobile apps are getting more complex every day. Users want more and more capabilities to complete their work quickly and enjoy their lives more fully. As software and mobile apps add features, they grow and require more time to fully test.  If you automate software testing properly, better software and happier customers will result.

Increasing the number of customers that want to use your software is fantastic! Even better, they are actively requesting more features – that means they intend to stick around rather than bail out and use another app.

Watch Video or Continue to Read

But, this growth in features comes with some distinct challenges – here are just a few related to your software and mobile app.

  1. Your software/app becomes more “crowded” with features, links and buttons – simplicity is usually a strong market driver and this complexity has to be managed carefully.  How will you increase features and still maintain a usable application?
  2. Your customers need support for a wider variety of operating systems, browsers, devices.  Again, this a great news but a big challenge to your development team as each operating system, browser and device has “little quirks” that are unique to it.  How will you know?
  3. Your software testing team becomes overloaded.  Not only because of the added features but also because of the number of operating systems, browser and devices that need to be tested.  How can all of the possible combinations be tested?
  4. Your developers begin “refactoring” the code to improve maintainability.  What? At some point in the life cycle of a software application, the developers will begin to recognize repeating patterns and behaviors within the app.  To make it easier and less time-consuming to maintain and improve in the future, the developers will begin replacing several unique code items with a single common function.  This is great and definitely improves both the software application and time to maintain, but it can also break things.  This “refactoring” happens all the time and is the leading cause of functions “breaking” when they worked previously.  Only detailed regression testing will find when this happens.  Automating software testing helps the regression testing go faster and at a lower cost.

Intelligently Automate Software Testing

What does this mean? Very simply, it means that you will need to improve the effectiveness and efficiency of your software testing team.  Here are some tips to help you improve the effectiveness and efficiency of your software testing team.

  1. Make sure your requirements and test cases are TIGHT!  Take a little extra time to make sure that you have accurate requirements with specific test cases.  This prevents much uncertainty when building and executing your software testing plan.
  2. Manually test the newest features.  New features are subject to change.  Older features tend to be more static.  Manually test new features until they stabilize enough to invest in test automation.
  3. Use Software Test Automation to handle the majority of your regression testing for older, more stable features.  A best practice is to treat your software test automation effort just like you would a software development effort.  You will reduce risk and improve your likelihood of achieving your goals if you take this approach. Tips on maintaining software automation testing suites.
  4. Hire a Software Test Automation expert to guide you through selecting the best software test automation tool set and strategy.  The learning curve for test automation is steep and expensive.  A proven, experienced expert can help you quickly select the best tools, strategy and plan to achieve your software quality and customer satisfaction goals.


Mistakes to Avoid when you Automate Software Testing

It’s easy to get in a mess when you automate software testing.  Here are some of the problems we’ve seen when we were asked to fix a “broken” software test automation system.

  1. Pick the Wrong Software Test Automation Tool – selecting the primary software test automation tool (or platform) is challenging.  You’ll need to compare your current technology stack AND future application/technical growth plans with the very long list currently available tools.  Hint – don’t automatically select the one your QA Team has used before – it might not be the best for your situation.
  2. Just Start Writing Scripts – Test Automation Scripts are the fuel that is fed to the test automation engine when you begin to automation software testing.  If you fail to use test cases and a test case management system, you run the risk of building an un-maintainable system to automate your software testing.

At TESTCo, we have been helping companies automate software testing since 2003.  Let us help you pick the best path to achieve your customer satisfaction and software development goals.

Software Automated Testing for Business Risk Mitigation

You worry about your business every day. Do you worry about the software that runs your business every day? Testing the software that makes your business run makes good business sense. You give employees performance reviews. It’s time to test the performance of your mission critical software too. In many cases software automated testing can be used to ensure top performance.

Watch Video or Continue Reading

If you’re in business then you almost certainly have software of some kind. It may be a simple store front eCommerce application or it may be a large and complex order processing system. Either way, I bet there are times you wonder and worry about your software. Will it work okay for everyone all the time?  Will it crash under a high load of customers? What if a new prospect can’t get what they need?

But, what if you didn’t have to wonder and worry? What if you could know, at any time, how your software is working and performing? What other (better) purposes would you use your thinking time for?

Most small and medium sized businesses do not invest in software quality. Most large companies invest heavily in software quality. The failure rate for small and medium sized businesses is approximately 50% after 5 years. The failure rate for large companies?  Almost Zero! Is investing in software quality the cause for this?  I doubt it. But, consider that a broken ecommerce or order processing system can definitely cause your business to fail.

We’ve shared a lot about what we’ve learned about this type of software testing. Here are more of our thoughts about software test automation.

Manual and Automated Software Testing for SMBs

Generally speaking, there are two aspects of software quality that are important to small and medium sized businesses.

  1. Does it work right when it is first built?
  2. Does it continue working properly as time goes by?

Item #1 can be addressed by using a dedicated software testing team to work with your team during the time you are building, improving or upgrading your software.

Item #2 can be address by using automated software testing that monitors the performance and accuracy of your software over time as you make enhancements and take upgrades.

The typical budget for investing in software quality using either functional software testing or software test automation is about 15% of your total project budget.

From a risk premium standpoint, 15% may appear to be large, but when you consider the consequences of a software failure and losing business during the down time, it may be a worthwhile investment.

If you wonder and worry about your software and how it will impact your business, please contact us and we’ll be happy to share what we’ve learned over the past 13 years of automated software testing, manual software testing and outsourced quality assurance.