Software Test Automation Case Study
Journyx has customers worldwide and have been improving their products with software test automation for the past seven years. The company offers customers project, time and expense tracking software, and resource management software for forecasting projects, budget status, and employee time and availability.
Challenges/Pains
Before hiring TESTCo to manage software testing and develop best practices, Journyx was in a world of hurt. Early on, they used company staff and even some family members to bang on new software releases, but that didn’t work so well. This was about 15 years ago. “It’s difficult, repetitive work,” said Journyx CEO Curt Finch. “No one really likes it, and it’s not really a core competency.”
The company wasn’t able to hire more people for testing, but they were always under pressure to release new versions of their products. “We couldn’t get a handle on when our next release would show up and what level of quality it would be. We could say we’re going to ship this come hell or high water by May 1st, but we’d be in danger of shipping broken software.”
“Back then, customers hated us and wanted to shoot us,” said Finch.
Journyx’s director of product management, Meredith Zachritz, had to manage a lot of the chaos at the time. “In the early days, our developers were responsible for testing their own code,” she said. “There were no robust peer code reviews. It was very messy. We just about drove ourselves out of business.”
The company didn’t have an official product manager back then, and they didn’t have the budget to hire testing resources in-house. “I was spending more than 50% of my time on testing and related fire drills,” said Zachritz. “It affected customer retention, and we lost confidence in our product’s ability to handle customer needs.” When mobile apps came along, complexity increased, and it was even more difficult to keep up.
Decision
The Journyx team eventually decided on outsourced testing, but they struggled with how to do it. Initially they hired contractors in India, but that failed. “We tried a couple of different firms,” said Zachritz. “It wasn’t apparent that they had QA expertise. If they did, the language barriers and time zone limitations were too difficult to overcome, and we couldn’t make it work. Our QA didn’t improve, and they didn’t understand what our goals were. They had no certifications or formal expertise, and no one was managing the entire project.”
Journyx ended up with nothing to show for their efforts – no test scripts, documentation or processes they could hand over to their internal development team. “We got what we paid for,” said Zachritz.
After that fiasco, Journyx hired a dedicated employee to be QA manager. They built up a set of test scripts – nothing automated – and some manual testing methods. “We had to train her,” said Zachritz. “She was only one person. When she was out, she wasn’t doing QA. And when she found a more attractive offer, she left. When you have someone in house, those are always issues.” That employee didn’t offer up any strategic guidance or best practice recommendations either.
Local, Accountable and Low-Cost Solution Helps Drive Revenue
Journyx eventually found TESTCo and solved everything in one fell swoop. They found a local company with the expertise to manage complex software testing and low-cost offshore testing labor. “What I like about TESTCo is that they’re an intermediary to different cultures,” said Finch. “At one point they switched testers from Romania to Ukraine, and we didn’t notice the change.”
“They’re here in Austin, and I can take them to lunch and hold them accountable or beg for forgiveness if we’ve messed something up on our end,” he continued. “We trust them, and the costs are very reasonable for what we get.”
Finch categorized the value of what TESTCo brings in terms of revenue, cost and risk. “TESTCo helps us understand exactly what we’re shipping,” said Finch. “We know customers will be happy with it right away, and our sales people have confidence in the software. That translates to increased sales.”
“The sales team is no longer afraid to demo stuff in presentations,” added Zachritz.
“Our costs are reduced because the testing talent is in another country where it’s less expensive to employ them,” continued Finch. “If I were to replace TESTCo with people here in the U.S., I’d have to spend about $5,000 to $10,000 more per month.”
As Journyx rolls out new product features and new software, they reduce risk by leveraging TESTCo resources. “For one thing, I have reliable, detailed insight into the status of everything,” said Finch. “With TESTCo we’re able to have a known level of quality in a consistent, predictable way.”
Journyx can also examine potential costs and complexities when considering the addition of new software features. “We can ask TESTCo how much it’s going to cost to add something to the testing,” said Finch. “That’s a double-check on the complexity question. It’s like a reverse estimation tool.”
Virtual In-House Team
Zachritz meets with TESTCo for bi-weekly progress calls and has daily email status updates from the TESTCo team. “We coordinate as though TESTCo is an in-house team,” she said. “The TESTCo QA process is run for me. I don’t have to do anything but review status and answer questions. I contact them with adjustments. They come to me with best practices and next steps guidance. All of that is very powerful for me.”
She now spends about 5% of her time on testing chores (as opposed to the 50% she mentioned earlier). “Now I can spend my time on planning and execution, and avoid all that fire fighting,” said Zachritz.
With TESTCo, Journyx can also scale up and scale back resources depending on their release cycles. “We can really manage it closely in terms of what we need at a given time,” said Zachritz. “And it doesn’t hurt us when someone takes a two-week vacation during testing. The TESTCo replacements are seamless.”
Software Test Automation Moving Forward
The TESTCo solution allows Journyx to automate software testing processes, as well. The goal is to reduce regression testing cycle times and get new releases out in a shorter amount of time.
Initially, Journyx didn’t tackle software test automation when starting out with TESTCo, but they eventually decided to move ahead with it in order to further enhance productivity and free up resources. “We tried to implement an automation suite earlier,” said Zachritz, “but coverage was bad, the platform was difficult to maintain, it was open source, and it wasn’t possible to upgrade it.”
TESTCo recommended changing out the platform. The two companies went through a new platform evaluation process, and found a new system that fit Journyx’s budget.
“We ended up automating the most time consuming manual tests and regression test cycles,” said Zachritz. “That speeds up our release cycle and in effect drives sales.”
The process helps them improve code quality, close the gaps between fixes and generally move faster as an organization. “Our support teams can now work on bug reports and patching custom solutions for customers,” she continued. “They can also work on the videos they create to help people configure their solutions. Software test automation allows us to work on more strategic projects because of the quality of releases.”
“All of it has transformed our business,” said Zachritz. “It’s really incredible how it works.”
Leave a Reply
Want to join the discussion?Feel free to contribute!