Can test automation tools replace the human testers?

Vincent: “I keep hearing a lot about test automation tools. In one of those SQA Forums I heard of companies firing their entire manual testing team citing the reason as a shift to automation. That makes me uncomfortable as I am still working mostly in manual areas of testing. Are manual testers going to be obsolete in near future due to test automation?”

Jolly: “I have been working as a test engineer for past 2 years and now I want to move into test automation to avoid being singled out due to lack of tool knowlege. I find lot of tools in the market like QTP, Rational Robot, Silk Test, LoadRunner, JMeter etc. I am confused where to start. Can you help?”

As you must have already figured out, these are couple of FAQs from my mailbox. But before I can start writing anything on this delicate topic, let me start with asking myself few related questions aloud:

1. Can automation tools ever *replace* human testers?
2. Can testing tools catch defects/bugs *automatically*?
3. Can tools ever develop enough (artificial) intelligence to think like a human tester?
4. Talking about scripted regression test tools (like QTP, PTQ, WinRunner, LoseRunner, LoadRunner, UnloadRunner, JMeter, KMeter, Rational Robot, Irrational Robot blah blah blah), which operate at the GUI (not unit) level. Is their valuation/pricing/license cost fair enough to provide you a healthy ROI (Return On Investment)?
5. Is it a question of Manual Testing Vs. Automated, or something much more complex and intricate?

Before we can come to a conclusion and start answering these questions let me come up with some facts about the contenders here: automation tools and human testers. I am a human being not a robot. So I will prefer starting with the human testers. Here we go.

What is manual testing? I think the term “manual” means, “involving or using human effort, skill, power, energy, etc.” Going by this understanding, any testing that requires the human intervention can be considered as manual testing. Let us see the areas where human testers can fair better than the automation tools:

Manual Testing Pros:
1. A human can think. She has the ability to think on her own and make decisions. Though decision-making is possible in case of computers (automation tools), they are yet to reach the level of (artificial) intelligence to beat a human being when it comes to decision-making.

2. A human tester can explore new applications using her intuition and self-learning abilities. Even if she has no experience of testing anything similar before, if she is skillful she can be able to quickly learn, explore and test the application. An automation tool might not!

3. A human tester can think out side of the pre decided test boundaries. In my knowledge, more defects are unearthed using out of the box exploratory testing approach than running automation scripts. Automation tests can only find you bugs for which they were written. They can’t find you any NEW bug. If you want to find new bugs, then you need to either find those yourself manually (that’s right, by a human tester), or write NEW automation scripts to find them.

4. A human tester needs no less spoon-feeding as compared to a tool. Computers are dumb, and so are automation tools. You change a variable/object name in a line of code of your AUT (Application Under Test) and the tool would struggle to figure out where it has gone! More over tools need step-by-step instructions to perform a specific testing task. They (tools) are poor at understanding natural human language. Imagine asking a human tester to test a “User Help Manual”. What if we are to automate this test? Can a tool test a human readable help manual efficiently? In case, you happen to know of a way to automate this process, kindly let me know. I am in search of such a tool.

5. Human testers rock when it comes to Usability testing where automation tools fail miserably. Tools are poor in judging what is more usable from a human point of view like screen alignments, appearance of windows, smoothness of object alignments, color combinations, ease of usability, an entertaining user experience etc. After all we develop software to be used by human beings, not computers! So who would be a better candidate to judge the usability of such an application? You already know the answer.

6. Human beings are good examples of being adaptable. We could adapt to the changing environment and survived when big creatures like Wooly Mammoth and the Dinosaurs perished and became extinct due to their lack of adaptability. Human testers can adapt and learn from past experience. But (automation) tool may fail to learn on it’s own. The entire suite of test automation scripts may soon start failing if a simple thing like an object name has been changed/renamed in recent code refactoring. A tool cannot remember and learn from past experiences to adjust it accordingly. Human testers can.

7. Considering the huge cost of License fees of the so-called GUI automation tools, human testers are still cheaper. More often than not you might run into a project where the ROI (Return On Investment) for such a tool would be in negative as compared to when the tests were done manually by human testers.

I think that is enough of pro-manual and anti-automation rant. Before you (wrongly) classify me as an ardent opposer of test automation, let me clarify that I am in fact a great supporter of test automation, if done rightly! If you have read my earlier post, then you might already know that I love to test in agile environment like TDD and as you know no agile development can be imagined without test automation. What I oppose is the idea of automating tests at the GUI level using commercial tools that eat up a large chunk of your testing budget. IMHO, this is just waste of money on something that could have been avoided if proper testing (including automation) was done at the unit level, in the first place. If you must automate your tests at the GUI level, then find out a way to develop and use your own test driver. This way you can save on a lot of testing budget, which could be spent on other areas of testing. e.g In our organization we use our own in-house “GUI Test Driver” to perform automated GUI tests, which uses XML and XSLT and runs on top of Nunit framework. But at any rate, I am a clear supporter of test automation if done at the unit level. And of course there are areas of testing (like stress testing, performance testing, penetration testing, link testing, API testing etc) where the need of automating your tests becomes an absolute necessity.

Test Automation Pros:
1. Computers work much faster than a human tester and are less prone to get confused while doing multi-tasking (doing several tasks at the same time, switching between them). Computers never care to spend time on attending a phone call or attending a review meeting/presentation, unlike a human tester. This can result in much more productivity as compared to a human tester, of course under certain contexts.

2. Computers are great at mathematical calculations. Computer's memory is much more accurate than a human brain, it’s capacity to remember stuffs is much more than a human and it’s capacity to retrieve data from stored memory is much faster and more accurate as compared to human brain. This accuracy of computation can be exploited via test automation, especially in tests that involve high volume of computation work.

3. Computers never skip any hours of testing. People get tired, people get distracted, but computers are great at repetitive tasks humans are not very good at. Computers need no rest, shows no sign of fatigue, never gets bored with repetitive work, can work on weekends, holidays and even night shifts without complaining at all (of course until they encounter a break down/malfunctioning). This power of computers can be used to our advantage via test automation.

4. Computers never complain about salary hikes, number of paid leaves, holidays. Nor do they ask for change of project if it starts to get boring working on the same project for a long time. Tools lack emotions and it can prove to be a good thing when looked from an employee satisfaction point of view. [Remember, an organization spends on a tool. Hence the tool is also like an employee working for the organization]

5. At certain contexts, test automation can be cost effective than manual testing. Paying for the development of a tool and a small group of testers to run it may be cheaper at contexts, as compared to maintaining a whole big group of testers testing everything manually.

6. When it comes to load test, obviously automation is an absolute requirement. Gone are the days when you summoned 100 employees into a big lab, asked each of them to hit the Enter key at the same time and hoped that they hit it exactly at the same point of time (millisecond level)! In testing we need accuracy of actions. And for scenarios like load and performance testing automation tests can offer us such precision.

7. Going for test automation can be a good choice for regression testing nightly builds. If you have a set of tests that you need to test periodically over a long period of time and if you are confident that little or no major change would be made in areas covered by those tests, then you might opt to automate such tests to save you some precious time.

Having said all this let me summarize my opinion. Coming to the core question of “can automation tools ever *replace* human testers”, I honestly don’t think that it is going to happen ever. Well, not at least in next few centuries, not before there are no more software to develop and they are *completely* tested or not before human beings become so lazy that they even stop moving around [watched WALL-E?]. Until then, I think it would be impossible for manual testing to be phased out. There will always be a requirement for *skilled* manual testers. You can't automate everything and you would always need testers with skill and experience, people who can think as they test. Automation needs a human tester to write the test scripts. Technology is yet to reach a stage when tools can themselves write tests on their own. And again automation may only speed up execution but what about the analysis of failed tests! You still need an analyst (a human tester) for analyzing a failed automation test. Furthermore, what an end user does can be compared with repeated testing and retesting of the application. And those tests are *mostly* MANUAL (unless we are talking about a piece of application like an API). So if you wish to mimic end user’s (potential) behavior then manual testing is vital for your test coverage. Automated tests mostly run on a constricted set of parameters. There is usually no guarantee that the situation at the end user’s will fall neatly into this constricted set. There is usually no way to predict such differences (in an automated way at least). It takes a (human) tester's imagination and creativity to cover these situations. So I believe, “Automation testing tools can never replace human testers”.

Does this make automation look like a poor contender? Not necessarily, I would imagine. As we have seen above, test automation has some Pros that can be used along with manual testing to derive a higher productivity and efficiency. The key is probably to have the right balance. Modern day cricket is all about having all-rounders. And IMHO, so is modern day testing field. I find it pleasing to discover myself a born optimist. Though people might say that too much of optimism is not good for testing, I however believe that optimism still retains the key to success. Let me try and combine the Pros of both class of testers (human and computers) and see if we can bring out their best:

1. Automate tests that involve repetitive, tedious and high level of computation work.
2. Let human testers do tasks that are highly intellectual and needs analysis.
3. Use automation for areas that are least prone to change and yet require frequent testing. This way you can save some energy on tests that need to be tested periodically.
4. As much possible, let humans do UI testing.
5. Use automation for areas like regression, load, link (broken link testing), API testing, first level agile (unit) testing.
6. Don’t automate if you know the UI is going to have a revamp. The slightest changes in UI can cause automation to fail. When you know the UI has been settled down, excluding any fixes to bugs, that's probably the time to automate.
7. Realize and understand that not all tests can/should be automated. It will initially take longer to develop effective automation scripts than to do manual testing. Knowing what and when to automate is critical.
8. Test automation is a supplement to help/boost the overall testing. Automation can neither suffice on it’s own nor it can replace other kinds of testing.

I feel that manual and automation testing should be allowed to marry to live happily ever after and to give an offspring of new hybrid tests. Why not take advantage of both of their Pros and create a win-win situation for us rather than fighting over manual OR automated battle? What do you think? Do you think automation tools are going to prove weapons of mass destruction for human testers? Do you think tools can never take over the place of a human being? Let out your opinions via commenting.

Happy Testing…

Share on Google Plus

About Debasis Pradhan

Debasis has over a decade worth of exclusive experience in the field of Software Quality Assurance, Software Development and Testing. He writes here to share some of his interesting experiences with fellow testers.