By Martin Bliss
Senior Developer, Technology
From Tactical to Strategic:
Scaling Quality to New Heights
Regression testing has never been more important than it is today. Between browsers and mobile devices, there are countless ways to access applications, and users expect the same experience on their phones as they do on their computers. In products where the user experience spans more than one device (Apple iCloud, for example), expectations for consistency grow even higher. QA professionals are uniquely suited to validate these expectations, but there is no getting around the fact that regression tests take time and can interrupt the cadence and success of a sprint. Even for teams employing source code versioning strategies, long-running regression tests threaten a team’s undivided attention to new work. The usual answer to this challenge has been to add more testers in order to shorten regression tests, but it’s no secret that increasing the headcount doesn’t scale; larger teams are met with increasingly complex challenges of their own.
Large team experience aside, software architecture can also pose challenges for regression tests: modern products contain complex moving parts, including cloud services, local data caching and intermittent network connectivity. Such elements constrain the number of ways a regression test can be divided, resulting in diminishing returns as the head count increases. To stay ahead of today’s device proliferation woes, software development teams need a scalable solution that blends the benefits of human analysis with the diligence and speed of modern computing.
We can find an answer through technologies designed to emulate human input. By producing the same interactions as a user (mouse clicks or key presses, for example), “test programs” leveraging UI automation frameworks can be made to automate the same tests previously performed by QA professionals with speed and precision. Augmenting QA teams with programs yields several key benefits:
Increasing Return on Investment
Because they are conducted manually and take hours or days to complete, regression tests are typically performed no more than once or twice per sprint, and usually include streams of changes from multiple developers. If defects are found, their origins are not always clear. Automation allows developers to get results iteratively within their personal development environment, gathering snapshots of test results in between code changes in order to support troubleshooting defects or documenting system dependencies without added headaches of sifting through other developers’ changes. Increased visibility of potential regressions at the developer level also shifts responsibility and accountability back to the individual: developers are able to see how their code quality affects the project instantly, allowing them to learn and grow in much faster iterations than before.
Test suites grow with each passing sprint as new tests are added for stories that are delivered. Only so many tests can be manually exercised in a day, however, so growing test suites either demand more QA personnel in order to maintain a return on investment in earlier sprints, or tactical decisions are made on which tests can be ignored for a given deployment, thus negating the ROI on some of the suite and introducing risk of potential defects. Test automation breaks through this artificial limit, ensuring that proper regression tests are not handicapped by resource constraints.
Increasing Team Productivity
Developers typically move on to other tasks in the same sprint while a regression test is underway or they get a head start on the next sprint’s work. If their previous work passes regression testing, all is well and good, but if the QA process finds defects, the developer will need to switch contexts. Automation encourages developers to finish what they’ve started before moving on, optimizing productivity with focus and keeping source control branching to a minimum. Automated testing fits well into existing best practices because developers are generally encouraged to keep their quantity of branches small and short-lived anyway.
Increased Focus on the User Experience
Testers are known for their discipline in executing consistent steps repeatedly in order to spot regressions in functionality, but there’s a higher calling often unaddressed due to time constraints: testing the user experience. With so many rules and requirements governing how a program should behave, it’s no surprise testers spend most of their time looking for regressions in logical behavior, leaving little thought to the user’s experience. It might seem logical that a functioning product is more important than a beautiful one, but the mobile app industry is learning that such a statement on product quality is too polarizing: users are more forgiving of occasional bugs than poor UI design. An app that is misunderstood or too complicated isn’t given a second thought in today’s modern era of mobility, especially when several alternative apps are just a few button taps away.
Preventing design flaws and user confusion is a responsibility uniquely suited for QA professionals where human judgment and experience are of greater importance than the correctness of a button’s click behavior. These professionals can cover a lot more ground focusing on subjective standards over objective ones, ensuring your product stands out from the competition. Automated regression tests are key to shifting quality expectations from a tactical focus to a strategic one, enabling developers to hold themselves accountable for their work.
As with any solution, it’s important to note that not all tests are candidates for automation. For example, tests for features that span multiple systems are especially vulnerable here. User registration could be difficult to automate if it has an email confirmation component, since most browser automation tools do not have native email client capabilities. Any feature that depends on third-party vendor integration points can also be difficult to automate, especially if it requires data stores that cannot easily be reset with each regression pass. For these reasons, automated testing cannot replace all quality measures, but it can be a star asset in any quality strategy.
Just as some tests don’t lend themselves to automation, not every project lends itself to the value of QA automation, either. We need to do additional architecture planning and integrate this QA automation process into Continuous Integration pipelines for full effect. If you’re building a small tool to make your own day job a little bit easier, the initial time sink and added complexity might not make sense, but consider the consequences of uncaught defects making their way into a production environment: if your product is intended for wide distribution (such as an app store) or there are a lot of moving parts and/or people on the project, automated QA could be the difference between enjoying barbecue with the family or being stuck at the office reading application logs on a Saturday.
This is the first part in a series on QA automation, and how projekt202 crafts quality and confidence in software development. Keep an eye out for part two, which focuses on collaboration between developers and QA, offering ways to reverse the stranglehold on velocity in order to unlock higher velocity.