What is a "Hard to Reproduce" Bug?
Well, you might be more familiar with the term "Non Reproducible" bugs! But I prefer using the phrase "Hard to Reproduce" over "Non Reproducible". Because, I belong to the category of testers, who believe that every bug is reproducible and if something happened once it can happen again! It depends on how much time and effort you are ready to invest on the investigation to reproduce the bug.
Anyway, as you might know already, these are the bugs, which are not reproduced every time even by following seemingly same set of steps/procedures. Every tester likes to find bugs. But if it is a "Hard to Reproduce" type, then it could easily become a nightmare for the tester too! But let’s face it. Every tester faces such bugs during testing. As a tester, quite often I am asked to reproduce some "Hard to Reproduce" bugs, reported by other testers or users of the application. And I believe that some of you might have faced such situation too. Here, our aim should be to try and get more and more information regarding the bug (who knows in the process we might reproduce it too!). These are five points, which I tend to adapt while hunting for a "Hard to Reproduce" Bug.
1. Always be on your toes; be alert! – By this I mean that you should be observant and keen for details while trying to corner such a bug. Keep your eyes and ears open for any possible suspicious looking process going on in the system. Look for those tiny little changes like flickering of the status bar, a missing button on the toolbar, a partially loaded dialog box, an empty dropdown list, slow to respond database queries and so on. If you happen to miss such a change in state, it might prove vital for your bug hunting!
Anyway, as you might know already, these are the bugs, which are not reproduced every time even by following seemingly same set of steps/procedures. Every tester likes to find bugs. But if it is a "Hard to Reproduce" type, then it could easily become a nightmare for the tester too! But let’s face it. Every tester faces such bugs during testing. As a tester, quite often I am asked to reproduce some "Hard to Reproduce" bugs, reported by other testers or users of the application. And I believe that some of you might have faced such situation too. Here, our aim should be to try and get more and more information regarding the bug (who knows in the process we might reproduce it too!). These are five points, which I tend to adapt while hunting for a "Hard to Reproduce" Bug.
1. Always be on your toes; be alert! – By this I mean that you should be observant and keen for details while trying to corner such a bug. Keep your eyes and ears open for any possible suspicious looking process going on in the system. Look for those tiny little changes like flickering of the status bar, a missing button on the toolbar, a partially loaded dialog box, an empty dropdown list, slow to respond database queries and so on. If you happen to miss such a change in state, it might prove vital for your bug hunting!
2. Trust but Verify; be suspicious! – Never trust anything without verifying. Take the bug report (submitted by other tester or whoever), just as a reference. Don’t follow it as a word from bible, or it won’t be long before you would find yourself moving on a wrong track! Do it yourself and see whether things are working as they are supposed to. After all we testers get paid to be skeptical. And finally the client is going to thank us for being so!
3. Look for Patterns! – Bugs love patterns. It’s the tester who has to identify the patterns to identify the bug. While trying to reproduce the bug, look for the possible patterns that are similar to a cousin of this bug. Who knows you might be able to find a pattern which could help you to reproduce this bug!
Here I would like to share one of my own experiences. Once I was trying to reproduce a bug related to a “database general error”. I was trying for long time now but was not able to reproduce it. Then I thought of searching for some similar bugs which might have been logged by other testers in the bug tracker. And in fact I did get a similar bug which was also related to “database general error” but in a different area of the application. Going through the bug report suggested that it was a bug which could be reproduced when Unicode characters were used in input. This was an important piece of information. I tried using few Unicode characters in my input string and bang! The application crashed telling about a “database general error”. Hurray!
4. Note down the things you are doing while testing! – Make a note of the things you are doing while trying to reproduce the bug. Include things like the test environment, other applications that might be running simultaneously, database configuration, the test data you are using, other instances of applications that might be using your AUT (Application Under Test) etc. Noting down these details can help to reproduce the bug. If you are able to get the bug but have not noted down the things you did to get the bug, then it might be difficult to recollect the exact sequence of things to reproduce it again.
5. Have patience! – Patience is the key while trying to reproduce a “Hard to Reproduce” bug. As a tester you have to show lot of patience while hunting for such a bug. These bugs are not easily reproducible and hence sometimes may make you impatient.
If you know any more methods that could help while hunting for a “Hard to Reproduce” bug, please feel free to share with me by leaving a comment. Venkat has written a nice post on a similar topic. You might like to visit it here.
Happy Testing…
Dear Debasis,
ReplyDeleteThis is MrSharma56.
Thats a fantastic blog truly resembles your experience.
All I want to add is:
1)The Severity and Priority plays a major role in searching for a non reproducible bug.
2)Developers help really matters in the case non-reproducible bug.
3)Project Manager should decide about the time to be given for a Tester to reproduce this non reproducible bug.
On a concluding note:
Every bug is reproducible but the case is that,it should be really a bug.
Please Correct/Comment me if I am Wrong somewhere.
Thanks & Regards,
MrSharma56
http://truetesting.blogspot.com
@ Tupaki,
ReplyDeleteThanks for you comment. When I say that I am assigned to try and reproduce a "Hard to Reproduce" Bug, I think it is well understood that, the Project Manager has taken the severity and priority into consideration and has done an analysis on whether this bug is worth investigating. But I do agree with you that developers can play a major role to guide the tester to reproduce the bug. But, unfortunately this does not happen(due to many reasons) in many cases. Moreover, sometimes the information provided is so unclear/incomplete that a developer can't do much about it to help the tester to reproduce the bug.
Thanks once again for your ideas.
Regards,
Debasis.
Pradhan,
ReplyDeleteWhile on the process of my regular activities, happend to locate a bug which did held a high severity from business perspective. I reported to the developer and explained him on its occurance and explained the implications which were likely to occur. But I was unable to reproduce the bug as it was inconsistent. This became one hot topic in a meet and came to an agreement that if i am unable to reproduce within a stipulated time, the developer would be forced to reject the bug and close it.
Though I worked hard to locate the bug in all possible ways I was forced to giveup and as a result, the bug was rejected.
Then, during one of my another task of tesing a new feature, it so happened that some kind of spark and clue arised in me to replicate the non-reproducable bug. With that clue as a base, I was able to work on it and was able to replicate that non-producable defect.
With this above experience of mine, I learnt that we always should have a relaxed mind set to always try to 'think out of the box' to solve any kind of issue. Not only for fixing bugs, any
thing for that matter.
Im not sure whether this would be a help to anyone?
I was able to raise a new defect in the same way.
Pradhan, do u like add something?
@ Kalpana,
ReplyDeleteThanks for sharing your experience here. I would have loved it more if you had described that not-so-easy-to-catch bug in more detail! Details like, the bug itself, it's mode of infection, how you could corner it, why it was so elusive, what made it a critical bug etc. Happy Testing...
-Debasis
Slightly off-topic and probably not reproducible:
ReplyDeleteIn my FF browser right now with multiple tabs your tab title says "My Top 5 Ways To Reproduce..."
Slightly off topic but hopefully worth a chuckle ...
Love your site.
jon
@ Jon,
ReplyDeleteGreat to see your comment here. Even in my FireFox browser with multi-tabs the title of this blogpost says "My Top 5 ways to reproduce..."! So we are at least clear that this is reproducible. But I wonder what made you think that this is a bug! What did you expect to see? Did you expect to see the title of my blog name (Software Testing Zone) instead? :) I am using a hack to display the post titles followed by the blog name. Hope this is clear now. In case you had expected to see "something else" in the tab title, kindly revert back with a comment. May be, we can discuss more on the topic. Thanks for visiting and leaving your comment.
Happy Testing...
-Debasis
Hi Debasis & Jon
ReplyDeleteMy add to:"My Top 5 Ways to reproduce..."
IMHO it isn't a bug, but an expected feature that FF provides for tabs.
I also tried to reproduce that issue.
It depends mostly on number of tabs currently visible.
And to achieve that quoted result there must be defined resolution of screen, size of fonts in operating system gui, even theme and css styles used for FF (maybe more things, but not tested)- the smaller fonts used for tabs, the more of them are needed to achieve that effect.
...But still kind of funny ;)
Best regards
Chris
Hi,
ReplyDeleteThis is a very interesting and helpful article. I am testing mobile applications where there are times when a bug is consistently reproducible in somedays, and sometimes the same bug doesnt seem to exit. Controlling environment parameters in mobile is not possible, GPRS being the main parameter. There are time when by the same set of steps a bug is reproducible, and sometimes (TO MY DISAPPOINTMENT) it is not. I have been facing this on and off in Mobile and WAP platform related bugs. Bugs on Desktop and Web are relatively more easy to re encounter.. Please let me know about any specific steps I must follow to lower down [or nullify] the non reproducible bugs count for Mobile and WAP. Currently most of them which are not getting reproduced are kept pending with lower priority. Should non reproducilble bugs be reported or not? Even when they are high priority bugs like crash?
@ Indira,
ReplyDeleteI am not a geek when it comes to telecom testing, but from your comment I concur that you are facing some bugs that are reproducible on a particular day and are non-reproducible on others. Here are few quick questions that might help:
1. When you say such bugs are non-reproducible on another day, is it on the same build of application or a different one?
2. Is any other related application running that might result in conflicts?
3. Why don't you try to narrow down the bug as much as possible (on days when it is reproducible)?
4. Can you get the bug investigated on a programmer's machine with a debugger on a day when you are getting it? Chances are high that if you are able to reproduce it on your case in that particular day then the developer just might be able to do so!
5. Think of any other factors that might be affecting the reproducibility of such a bug and let the developer know about it.
Should non reproducible bugs be reported or not? Even when they are high priority bugs like crash?
I personally believe that every bug should be reported irrespective of whether they are easily reproducible or hard to reproduce! By doing so we will have a record of things if someone else faces such a bug in future. Talking about priority, a crash (high severity) bug does not necessarily mean it should be of high priority. To read more on bug severity and priority follow the URL: Priority Report
Happy Testing...
-Debasis
Hi Debasis,
ReplyDeleteThanks for your prompt reply. Your comment for non reproducible bugs is most encouraging. These are the answers to some of your questions. Please do guide me...
1)When you say such bugs are non-reproducible on another day, is it on the same build of application or a different one?
Ans: Quite a few times its the same build, and quite a few times a different build, but the dev guys confirm that the code area where I claimed to find the non reproducible bug hasnt been touched.. If i cud find it then, i shud be able to find it now. Nothin has changed..
2. Is any other related application running that might result in conflicts?
Ans: No such application runs for our application..
4. Can you get the bug investigated on a programmer's machine with a debugger on a day when you are getting it? Chances are high that if you are able to reproduce it on your case in that particular day then the developer just might be able to do so!
Ans: Some bugs are device specific and cant be encountered in the debugger/emulator.. I do let the dev guys know about every bug thats been hit..
I do try to trace the bug to fullest when its consistently happening.. but it seems difficult sometimes to re hit.. doing R&D for such bugs often which helps me know if there are some factors which could lead to the mystery bug..
Please do let me know you comments on the same..
Thanks
Indira Pai
@ Indira,
ReplyDeleteYou said... Quite a few times its the same build, and quite a few times a different build, but the dev guys confirm that the code area where I claimed to find the non reproducible bug hasn't been touched. Nothing has changed."
I don't know why but my instinct warns me of the possibility of a Heisenbug in your case. In case you are new to such class of bugs you might want to read more about them here: Heisenbugs - A Tester's Nightmare!
Happy Testing...
-Debasis
Hi Debasis,
ReplyDeleteThanks for your prompt reply again...Yes it surely sounds like a Heisenbug case for me.. Your articles have really helped me a lot.. thanks for your guidance... though there is nothing much different in the build which the dev guys check the bug and in which i hit them GPRS is one big factor and the other factor being handset condition: memory, battery.. I will now keep an open eye to these factors too to see if anything is related to them or not.. Thanks again..
Every thing has done....But still not reproducible...
ReplyDeleteSo, in what status shall i keep this bug...??
It is good to have a useful information from your side. I am not a software tester, but still I can say after some experience with testing that each and every bug is reproducable with the passage of time. The more you go for the product maturity, the more you face bugs.
ReplyDeleteI am sure that your steps will be beneficial for many out there looking for a fix for bugs.