Test everything…or not.
A month ago Michael Bolton sent me an article that underscores the problem of “all tests being deterministic” and “we have to cover all the functions in this application.” Something I was thinking about while I read the article was how a checkup or physical at a doctor’s office works and how that compares to software testing.
Often stakeholders want testers to “test everything.” Can you imagine if instead of checking the usual health indicators and vital signs your doctor tried to test “everything.” Consider how long that would take, and how much it would cost. It would be ludicrous to do such a thing especially if you were in good health.
However the risk in not testing everything (and the primary fear of our stakeholders), is that the usual indicators and vitals don’t always turn up something in the examination room. A patient could leave and drop dead in the parking lot. So what can doctors, or in this case testers, do knowing this very real risk?
I think the key is having a medical history for your software. Having year’s worth of tribal knowledge about the application helps you decide what indicators and vitals matter. Having this history helps a tester perform tests that are likely to turn up something important about the application while it’s in the building. While there still is no guarantee that the application won’t drop dead in the parking lot, it does increase the chances of finding something significant before it does walk out the door.
Explore posts in the same categories: Software Testing
Benjamin Yaroch's Blog 
May 7, 2012 at 2:26 pm
Interesting points you are doing Ben. I agree with you on it as well. What I AM still wondering about is how the tribal knowledge and medical history is gathered, maintained and grown in a good and healthy way?
Especially I am thinking in terms of the qualitative information gained when testing, you know the information that is not really relevant at the moment as a bug or problem, but rather the kind of medical history and good to know things that come out of your testing. How do you handle this constant information flow?
May 7, 2012 at 2:42 pm
For me it has always been an oral history. For example the team will say “oh yeah, that feature breaks every time we make changes” or “we are very worried about feature X because…” Beyond oral history bug tracking databases can contain some interesting information; skimming titles over time can help build a history.
I’m cautious about formally documenting histories because it can be very burdensome. However having a watch list as a wiki or on a whiteboard could be helpful. With that said I’m sure others will have ideas.
May 7, 2012 at 4:21 pm
A terrific metaphor on knowing where to test. To take it one further, on systems that we do have more history with, we are better qualified to know were to focus our sessions, and with systems that are new to us we must be more careful and ask more probing questions up front.
May 7, 2012 at 9:01 pm
Good one Ben.
We need to pay close attention to the evidence, but at the same time try to think outside the square. You may have loads of IP on your software (medical condition) and diagnose the problem as being X because it matches that IP, when really the problem is Y.
The Drs said it was Bell’s, I’m assuming because all their previous history pointed them in that direction. Even as the evidence for Lyme piled up, they were so sure about their previous history that they disregarded the Lyme.
It’s great to maintain IP, but don’t always rely soley on that. More questioning, more investigation… test those assumptions.
:0)
May 7, 2012 at 9:08 pm
Indeed, sometimes the hoof beats you hear are zebras.