“If you cannot repeat what you are testing then you are NOT testing properly! Think of regression testing. What will happen if your regression tests are exploratory by nature and are non-repeatable”? A tester friend of mine argued strongly, trying to give himself ground to continue his fight against my advocacy for exploratory testing!This made me cogitate. Not that he had succeeded in proving that Exploratory Testing (ET) sucks! But I had to start thinking if repeatability (read reusability) of a test is really all that bad! I guess “repeatable test” VS “non-repeatable test” argument is just another face of the “Scripted testing” VS “Exploratory testing” debate. Here in this post, I am not going to continue the same debate trying to prove which one is better and which one is not. Rather I would like to think more on the repeatability aspect of a test!
Some questions to start with can be:
1) Can ALL the tests be made repeatable?
2) Should ALL the tests be made repeatable?
3) By making a test repeatable are we not (may be inadvertently) making them too predictable?
4) Can repeatable tests discover NEW problems (defects/bugs) in the system?
5) Blindly trying to make each and every test repeatable. Is it worth the effort and expense?
6) Why should we repeat just a set of tests (test cases, test scripts whatever), when we can utilize the same time exploring much more tests that may uncover unknown defects!
7) When we say a test is repeatable, is the test really repeatable in all its senses? Can someone guarantee that the test runs exactly in the same way (environment, concurrently running processes and applications, exactly same DLLs loaded at that moment, machine condition, configuration settings, and so forth) as it had run the last time around?
8) Do repeatable tests guarantee reproducible defects/bugs?
9) A repeatable test can make sure there is no re-occurrence of an earlier defect. Ahh well! But how long? Will these so-called repeatable tests not loose value over a period of time (after a number of iterations)?
10) Is software testing all about repeatability?
The argument in support of making your tests repeatable may hold good to some extent, in certain cases like Regression testing or for that matter Performance testing. But is there any point of trying hard to make ALL the tests repeatable? Here are few interesting quotes from some honorable Testing Gurus/Experts:
Highly repeatable testing can actually minimize the chance of discovering all the important problems, for the same reason that stepping in someone else’s footprints minimizes the chance of being blown up by a land mine.
- James Bach, Test Automation Snake Oil, 1996
- Brian Marick's talk Classic Testing Mistakes
By repeating tests we actually make sure we are avoiding other possible defects (remember minefield analogy? Stepping on someone’s footstep actually makes sure we may avoid stepping on a live landmine) thus minimizing our chance of discovering new defects!
Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual.
- "Pesticide Paradox", Boris Beizer, in his book Software Testing Techniques, Second Edition, 1990
"Pesticide paradox" compares software defects with that of pests! Imagine a context when a farmer applies certain pesticide to get rid of insects from his crop. There is every possibility that there will be some insects that will survive the pesticide. If he keeps applying the same pesticide, the insects eventually may build up resistance and the pesticide would no longer work! "Pesticide paradox" describes the problem that a regression test series gets less and less powerful as you use it over and over again. When a set of repeatable tests is run for a period of time they tend to pass more often than failing. A test is valuable when it fails and the failure uncovers a bug. Running a set of tests that seldom fail and have least chance of uncovering defects sounds like a bad idea. Isn’t it?
Having said this, does this mean repeatable tests are waste of time? May be not!
1) Think of Performance tests that a tester may need to run over and over again for days in and days out.
2) Think of Benchmark tests.
3) Think of Build Verification Tests/Sanity Tests that need to be run every time you have a new build!
4) Think of Regression tests.
5) Think of a scenario where you need to run certain tests that are important in nature. The tests that verify some very critical functionalities of the application and these tests must be run periodically to make sure those functionalities continue to work without issues.
So, it appears that having some repeatable tests can be helpful for your test project along with those tests that are non-repeatable. As always, it certainly seems to be context dependant and depends on your testing mission/goal! What do you think? Do you think ALL (what that can mean!) tests MUST be repeatable, as if we are not software testers rather some robotic human beings trying to repeat some algorithm already decided in advance! Repeatable tests! Are they the need of the hour or necessary evil? Your thoughts please.
Happy Testing…
Further Reading:
Reasons to Repeat Tests - By James Bach
Repeatability is Overrated - By Elisabeth Hendrickson
Hi Debasis!
ReplyDeleteNice Article. Regression and Progression are the two terms which we use in our company.Well Regression stands for the stuff which you already knew and explained.
Progression Testing is Testing the new enhancements or the new modules in the already existing application.(Over the years we are/will work on the same application/system :))
Now Automation vs Regression, not all Regression Test Cases can be automated. Eg: 1) Check the delivery of email when some event is triggered.
2) Check whether a report has been printed or not when the print button is clicked.
To conclude great continued effort of yours in knowing and making us learn new terms and ideas.
All the best. Cheers :)
Apologize for any spelling/grammar mistakes if exists.
hi
ReplyDeletethis is vinay ,a tester
Can u suggest come good blogs for software testing as well as some good books.
Regards
Vinay
@ Vinay,
ReplyDeleteI would like you to read an article that I had written quite sometime back - How to get started in S/W Testing. You can find some valuable testing resources to start with on that article.
Happy Testing...
-Debasis
Another nice article..
ReplyDeleteThanks Debasis for all Your articles. It is always nice to return to them and learn once again something that I missed. Keep Up that Good Work.
ReplyDeleteBack to the article.
As You know I'm still new in QA, so I could be mistaken in naming conventions of testing processes. You're free to correct me ;)
I use repeatable tests/test scripts in regression, and when it is possible. Usually they run in background and check if everything is ok in new build. They run usually few times, while i do rest of testing with new ideas. I think It is the best approach I could use.
Exact repeatability isn't an option.
ReplyDeleteWhen we're testing on a Windows platform, there are dozens of processes running, each in its own state, each with different allocations of memory. Those processes are interrupting other processes, being interrupted by other processes. Subtle differences in timing exist between any test and the next, whether the steps in that test are being performed by a human user or by a machine.
So it's time to think more clearly about this. A repeated test is one in which we are repeating some dimension or some factor of the test--presumably one that we think matters. Here are some questions to consider: What dimensions might not matter, even though we're paying attention to them? What dimensions might matter, even though we're not paying attention to them? To what extent are we aware of these dimensions? To what extent can we control them (or not)?
---Michael B.
@ Michael B.,
ReplyDeleteThanks for your insight on the topic of repeatability of a test! Truly NO test can be repeatable in it's true sense. I like your example of tests running on Windows platform and how it is *almost* impossible to recreate each and every underlying running states of the OS while trying to REPEAT a test! As you said, it's a matter of judgment on whether we agree to overlook certain dimensions of the test while we concentrate on certain dimensions, which we think might be important!
Thanks for enlightening me with your thoughts.
-Debasis