Test everything…or not.

A month ago Michael Bolton sent me an article that underscores the problem of “all tests being deterministic” and “we have to cover all the functions in this application.” Something I was thinking about while I read the article was how a checkup or physical at a doctor’s office works and how that compares to software testing.

Often stakeholders want testers to “test everything.” Can you imagine if instead of checking the usual health indicators and vital signs your doctor tried to test “everything.” Consider how long that would take, and how much it would cost. It would be ludicrous to do such a thing especially if you were in good health.

However the risk in not testing everything (and the primary fear of our stakeholders), is that the usual indicators and vitals don’t always turn up something in the examination room. A patient could leave and drop dead in the parking lot. So what can doctors, or in this case testers, do knowing this very real risk?

I think the key is having a medical history for your software. Having year’s worth of tribal knowledge about the application helps you decide what indicators and vitals matter. Having this history helps a tester perform tests that are likely to turn up something important about the application while it’s in the building. While there still is no guarantee that the application won’t drop dead in the parking lot, it does increase the chances of finding something significant before it does walk out the door.

AST newsletter article: “The Road Ahead…”

I wrote an article for the January AST Community News, you can find it here. Enjoy!

Being seen as a Tester: Not always a good thing

Let me start by saying that normally I’m proud to be seen as a software tester. I understand how challenging, rewarding, and interesting this job is. However, there is a dark side to being seen as a tester. Testers can be treated differently by people who don’t understand or don’t see testing the same way I do. This disconnect can not only devalue the testing effort but at its worst it can put an entire project at risk.

Let’s start with how I want to be seen as a tester. I view software testing as a service, and I want to provide valuable feedback to my stakeholders in a timely manner. To do this I need to maintain a high level of project awareness and knowledge at all times. This means I need to be on the inside of a project actively participating in discussions and learning as I go. I want to be aware of what’s happening in development, marketing, customer care, and even sales. By being actively involved at this level I gain a greater understanding of the context I’m working in. I use this improved understanding to increase the value and effectiveness of the testing effort.

With that said, often people view software testing as an administrative task that happens toward the end of a project. In their minds, a tester makes sure the software works the way it was specified and fills out test documents to satisfy that expectation. A tester is typically given limited information about the project and is usually only included in discussions if it’s felt that they relevant to testing (by their definition). This draught of information leaves the tester with little to no context, which can lead to feedback that has little to no value, which results in reduced overall testing value and effectiveness.

So while I love being seen as a tester for the right reasons, I would rather not be seen as a tester for the wrong reasons. To combat this and add value to a project I need people to think of me as a developer, marketer, support representative, or sales person. This ensures that as conversations occur or information flows I’m included and privy to it. By being included I don’t miss out on critical project information that may not on the surface seem “tester appropriate.” My philosophy is I would rather have too much information then not enough, when doubt invite me or share, then I can decide if its “tester appropriate.”

CAST 2010 transformations

I attended CAST this year in Grand Rapids and as usual it didn’t disappoint. For most returning CAST’ers the reason they come back year after year is the high level of interaction or conferring. And while that is very true of CAST what I find interesting is the transformations that occur in the matter of 2 days from this conferring.

Each year CAST attracts a number of new attendees that feel that software testing is really about being the sentry of quality. They view themselves as the gatekeepers of quality and often talk about standing in the way of shipping products. I’m always interested in seeking these people out, discussing this with them, and seeing how their opinions change by the end of the conference. Typically by the end of the conference they have shifted their focus away from quality assurance and toward quality assistance, which is amazing.

For me I love the fact that CAST provides an environment where testers can quickly learn how to advance the state of our industry trough interactions and informal discussions in the hallways, over dinners, and between sessions.

If you haven’t had the chance to experience CAST you really need to, especially if you’ve been testing for some time. CAST 2011 will be in Seattle Washington and is looking like it will be an amazing experience. Be sure to check out the promo video: http://www.youtube.com/watch?v=NpLcIFmmSF8

Firmware is software

I regularly work with hardware that interacts with the software (not off the shelf PC hardware, custom hardware device). Over the years the hardware I work with has gotten more complex, which also means the firmware has gotten bigger and more complex.

What’s interesting is that some people view firmware and software as unrelated and dissimilar especially from a testing perspective; not true.  As an electronics hobbyist and software tester I can tell that firmware is software, and software is firmware; they are one in the same. Sure software can be millions of lines of code but when you consider the size of a microprocessor, thousands of lines equals millions.  Further gone are the days of assembly language  programming, most hardware developers use C and some are starting to use C# (not sure how popular this is). So the PC and custom hardware worlds are starting to collide.

From a testing perspective I’ve found that any test that can be performed on the PC side can be performed either directly or through light adaptation on the firmware side. A good example is boundary and race conditions, which exist in firmware also and can create some very interesting bugs (but that’s for another day). Honestly I can’t think of much that is different between firmware and software from a tester perspective. To this end I think its unfair to assume that since its “just firmware” it doesn’t need the same level of attention or somehow requires a different skill set to test. In fact I think that many of the lessons we have learned on the PC side could be of great benefit to the firmware side.

I think the firmware and software testing worlds have collided, and we need to embrace it as testers. So please give firmware the testing attention it deserves, after all its software too.

Attend CAST 2010

Last year was my first CAST. I had heard countless good things from the community testers I interact with, this in combination to the cost savings were the main drivers for my attendance (employers like the fact that its 1/2 the cost of a larger conference).

Once I got there I discovered why it was so highly recommended. Sessions aren’t just a person lecturing and then you leave, you get the opportunity to ask questions and test the presentation. This takes each session beyond just a lecture and into true learning and sometimes a debate. In addition I discovered the best part of CAST happens after the sessions in the hallways.  I got to break bread and talk with testers I respect about my ideas, I got to ask questions, and share war stories. I also got to interact and network with testers beyond the casual hello while passing one another in the halls. Instead of a stack of business cards, many of which never get used, I was able to build a true “support network” which I use through out the year.

You need to attend CAST, but be careful its addictive, you will never view conferences the same again.

Attend CAST 2010

The 5th annual Conference of the Association for Software Testing

August 2-4, 2010, Grand Rapids, Michigan, USA

“Skills in Testing”

About CAST

CAST reflects the AST’s core mission: to build community amongst scholars, practitioners, and students for the advancement of the practice of software testing. In 2010, CAST aims to leverage peer collaboration to build an enhanced understanding of how various skills influence tester effectiveness.

CAST offers a unique opportunity to learn and confer with others that simply isn’t found at other conferences. Each scheduled session allocates time for facilitated “open season” discussions that encourage participants to question and challenge the presentation. What takes place in the hallways, at receptions, and during meals and lightning talks truly sets CAST apart; for many attendees, the greatest value is derived from the opportunity to discuss and delve into the topics that matter to them.

Space is limited Register Today!

More information and Registration: www.CAST2010.org

Full Conference Pricing (non-member)

$630 by May 15th, 2010 ($280 savings)

We can’t wait to see you in Grand Rapids!

Understand [figure out] how it works

I’ve always been intrigued by how and why testers think and do the things they (we) do. Each day I get the opportunity to see this tester uniqueness played out, and it is always interesting and often thought provoking.

Lately I’ve been thinking about the differences between new or novice testers and more experienced testers. This month I got an opportunity to witness some differences first hand. A couple of new testers had seen a bug but couldn’t reliably reproduce it. To further complicate things they could only get the issue to appear once in an eight hour span of time, drastically slowing their isolation efforts. When I got involved they had already spent days working on it and were mentally lost. They had gotten to the point where they were blindly searching with little to no direction and had a lack of tester motivation. While this wasn’t a project I was working on I decided to jump in to see what another set of eyes could do. I also wanted to provide some guidance, motivation, and support.

I started by asking lots of questions about what had been done, what they were doing when it occurred, and what they thought might be the root cause. I’ve found that in situations like these disciplined thought and structure can help clarify the  problem and provide new direction. I also started brain storming about tests that could be done to exercise each possible root cause. Just as we finished our discussion one of the testers announced he had reproduced the problem; the best part was he was able reproduce it at will. The group was excited that they had found repro steps and that’s where they stopped.

But that’s where I started. While it was great to have reliable steps that didn’t tell me what the problem was. In the end what the group had seen was just a symptom, they had not isolated or understood the underlying failure. I began thinking through the repro steps and comparing that to what I knew about the different parts of the application. I then headed over by the developers to discuss my thoughts and ideas. I asked a number of questions about what was happening during that span of time, I also confirmed what I thought I knew about the application and how it worked. I did this to try to better understand the how and why of  the observed symptom, with the intention of exposing not only the root cause but more importantly to uncover additional  areas that needed to be tested.

So what did I learn from this experience? I learned that I’m not satisfied with exposing a symptom I need to understand how it works behind the scenes. I don’t trust whats directly in front me, sometimes what I am seeing is a smokescreen or a byproduct of a different problem. As I test I need to understand whats happening and how it happens, my intention is to better understand the application as a system.  To do this I use multiple oracles and a number of different information gathering methods which include strategic testing, asking good questions, and thoughtful observation.

I’ve found that having a system level understanding  of the application you are testing opens up a whole new world of testing possibilities. Knowing how it works lets me dig deeper and find important problems early.

Do you understand [figure out] how it works?

Comptability isn’t a defect?!?

When did compatibility become something other than a bug (defect)? Especially when its a mission critical system with older devices in the field.

US Military GPS compatibility issue

Casino Bug

As testers we talk a lot about bugs and how they can/do reduce value in our products and projects. Well here is an example of a bug that incorrectly increased a monetarily value, which decreased the software’s value.

Woman Wins $20.18 instead of $42 million thanks to a software bug

Automation is like Duct tape…

It seems to me that an awful lot of people treat automation like duct tape.

Duct tape tends to be the miracle cure for all kinds of problems (there are websites dedicated to this premise). If the bumper on your car falls off “duct it”, if your plumbing leaks “duct it”, if you need a prom dress “duct it” (no joke ppl make prom dresses with this stuff). The problem is in most of these cases there is a better way to accomplish the task and the “duct it” method just leaves a sticky residue and doesn’t really fix the problem.

That seems to be the way people talk about (or use) automation. If testing takes too long “automate it”, if testing costs too much  “automate it,” wee want zero defects “automate it” (really there are ppl that believe that zero defects is possible). Here again automation is not a cure-all and in many of these situations there might be a better way. Be wary of the knee jerk response “automation it” you may just end up with a sticky residue that doesn’t really fix the problem.