Do You Really Need Automation

Do You Really Need Automation

If you've looked for a QA job in the last few years, one item inevitably shows up on the requirements list; automation experience. And not just automation as a fundamental concept, very specific experience with tools like Selenium, Cucumber, Capybara, Watir, QTP, and SilkTest. At this point, automation is basically a ubiquitous term within Software Quality Assurance. I would go so far as to consider automation a software industry sacred cow of sorts when in reality, adds no real value for many of the companies who use it.

That said, most organizations COULD benefit from automation if automation principles are properly applied. And therein lies the first issue. Very few organizations seem to understand how to use automation where automation will yield the most value. As an astute reader will note, many of the tools listed in the first paragraph, and in most job posting requirements, are UI automation tools. Interestingly enough, in practical application, UI automation generally adds the least value of any type of automation. Why? One word; Fragility. UI automated tests require a ton of maintenance for a very basic reason. UI typically has the highest volatility and changes more frequently than any other part of a software application. To complicate the matter further, most companies use an Agile development methodology where software evolves over a set of iterations and feedback loops. Yep, this means even more UI volatility, changes, and complete rewrites than traditional software development processes. With a majority of developers caring little about testability when "working code" is the goal, the QA team now spends time maintaining automated test cases instead of actually testing the product. Knowing how, what, where, and when to automate makes the difference between an unmaintainable mess of broken test cases that may be failing due to simply being outdated and a massive productivity booster. For most companies, neither the time nor the money exist to do it right so the age old adage of "do it right or don't do it at all" should be applied to automation and UI automation very likely isn't right for many organizations.

Cost plays a major part in the decision to start an automation process. It's a hell of a lot cheaper to write something once and run it every time a developer kicks off a build right? Yeah, not so much. That's an expression of Silver Bullet Syndrome. Automation does not magically save money, time, or resources. When done right, automation CAN in fact save money but hiring people with the expertise to add value through automation don't typically come cheap, even offshore. Realistically, 40% - 60% more than their manual tester counterparts in many cases. A worthwhile automation engineer often has skill sets that overlap the development group; an applied knowledge of a scripting and object oriented language, decent SQL skills, a firm grasp of system and data architecture to name a few, and that's in addition to having a mastery of QA concepts. Frankly, in my experience, very few people possess the entirety of these skills and must either be trained into or learn them.

And what happens when automation engineers do learn learn all of the required concepts? Employers in many tech markets today are struggling to hire and retain good staff so a good automation person eventually learns their value and wants more money since they now have a higher perceived value to your organization or are quickly poached given the current demand for the resume line item that comes with the automation skill set. Losing a critical resource will increase operating cost and/or leave a gaping hole in your org chart. If the organization is both willing to absorb the upfront cost, retain employees, and has software where a substantial ROI is gained from the use of automation, investing in automation can and likely will pay off. For everyone else, invest in a good Lead QA Engineer who can train up a few eager to learn n00bs and use learning and writing automated cases as a way to fill down time between projects or deliverables. You gain a double benefit by providing self paced career advancement opportunities for your staff while developing skill sets useful to your organization, all in time that would otherwise be spent updating fantasy football rosters or posting cat pictures on social media.

After taking into account the issues above, is automation REALLY worth it? Does automation actually increase quality? In most cases, the answer is no. Time spent writing and maintaining automated test cases distracts from actually testing software. Think about all of the test cases that are NOT good candidates for automation and how much time the team spends maintaining automated test cases. For many companies, it's a wash at best and at worst, actually leads to a DECREASE in quality. Projects with an extremely tight delivery window and no previous automated test cases are especially susceptible to negative net quality outputs due to incorrect usage of automation. Like anything else in business, organizations need to invest capital and time wisely to yield optimal ROI. Quality is no different. Consider again the prevalence of Agile. With the test cycle sitting at the backend of the SDLC (or iteration in the case of Agile), over complicated processes mandating or pressuring the use of automation distracts from the life blood of QA; testing. Unsurprisingly, the apparently lack of focus on producing "working software over comprehensive documentation" as the Agile Manifesto states exacerbates the issue greatly. To put it bluntly, automation just for the sake of doing automation sucks yet an an astronomical number of organizations do it and ingrain it into their hiring practices. Many will disagree and claim that automation increases quality, at minimum, by allowing easy regression. This is a valid argument and may be true for some organizations, but speaking from experience, quality WILL suffer with incorrectly applied automation techniques. If ample time exists, optimal automation processes have been identified, the quality staff has the technical chops, and/or downtime exists between deliverables, automation can lead to an increase in software quality.

Consider this very basic scenario. A company builds a website where users must login before gaining access to some function of the software. Super basic stuff that software testers deal with every single day. Seems like a perfect candidate for automation since login makes up a fundamental part of the software. Write once, never worry about it again. Of course, tons of free tools exist with a wide variety of companies using them so why waste good cash? Just use one of the free tools right? Every workflow, every boundary condition, every character combination, and every conceivable negative test case gets automated. And then the call comes. John from sales doesn't like the screen flow and thinks click through rates will suffer with the current login UI and legal realizes that your web software needs COPPA compliance so a 13 years or older age gate requirement needs to be factored into the login process, of course requiring additional screens and several alternate workflows. Those 50 automated test cases? Scrap em' or do some serious rewrites. Even the most robust modularized automation tools would take highly skilled staff ages to get back to a point where the automated cases add any value let alone the unmaintainable garbage resulting from many "free" tools. Just to add a bit of salt to the wound, the $10 an hour intern could have rerun the test spreadsheet regression cases in less time than it would take the $35-$50+ (in early 2010s dollars) an hour automation engineer to update everything. Throw in a tight timeline and barely tested software just went out the door. No bueno!

Maybe it's silver bullet syndrome, maybe its management by magazine, hell maybe its a little of both but this scenario, sadly, exists in most companies. While most companies will see a benefit from correctly applied software automation principles, many others are actually doing a disservice to themselves and their QA team by putting a heavy emphasis on automation. The keyword here is need. Could all companies benefit from from some well-placed, well-timed, and well-structured automation in their software projects? Absolutely. Is it needed? That depends on the company's commitment to doing automation right. The startups and small companies who make up the majority of the software producers out there just don't have the time, resources, and cash need to reap the benefits from automation. For them, the best bet is to start small and avoid volatile areas of the system. Think about repetitive tasks that eat up TONS of tester time, like generating data, and start there. For growing and larger companies, a good process holds the key to success. Each organization is unique so spend a lot of time upfront identifying where automation can add the most value or shore up gaps in the testing process where human based testing is difficult or impossible (performance or load testing perhaps). Treat automation like any other project and create a roadmap and backlog. How should a company determine what and how much to automate you may ask and perhaps, how to prepare to automate when conditions are favorable in the future? That is a story for another time.



Leave a comment