A few weeks ago, a colleague of mine tweeted an assertion that the context-driven testing school “censures” people who advocate for best practices.

I responded: “No one has the authority to censure people on behalf of the school” and “Surely we can disagree with people without censuring them.”

The discussion resurfaced on the “software-testing” listserv on yahoogroups, with statements like these:

• “Context-driven is important as an antidote to the nonsense that otherwise pervades the industry. Consulting companies the world over are promoting best-practices. But there *are* no best practices. Therefore they are lying. They are promoting cynical, self-serving practices, actually.”
• “I ‘know’ they are deliberately committing fraud for gain, as much as I can know anything in this industry.”
• “To say there are best practices is to say something that is not true, and that any competent person knows is not true, and that any incompetent person is not qualified to be saying at all. It is indefensible, except that people who try to defend it generally say that they DIDN’T REALLY MEAN IT (that’s what Rex Black once told me) which is to say they were telling lies.”

A few people on software-testing–most notably Fiona Charles–stated their disagreement with these generalizations. But not many. Some agreed. Most were silent.

I think that calling people liars for advocating a commonly-espoused point of view is beyond unreasonable. To me, if seems like hate speech: an unfair and inaccurate characterization of many of the criticized group, likely to stir up negative emotions and to reinforce negative stereotypes.

Context-driven testing is about finding ways to do excellent testing in the actual context of the project. We adapt to the project. That includes adapting to the views and practices of the people doing that project. To do this, it is a fundamental skill for context-driven testers to be able to listen sympathetically, find common ground, and work effectively with people who have other points of view. Slamming groups of people as liars is antithetical to this. It promotes closed-mindedness, which for a context-driven tester, means ineffectiveness.

The underlying disagreement

Suppose that Joe says that in a certain situation, using Testing Technique X is a “best practice.” What does this mean?

1. The straw man

One interpretation is that Technique X is the genuinely best thing to do in this situation. That we know this to be true because we have done the enormous amount of research that would be needed to demonstrate its superiority.

Unfortunately, we don’t do this caliber of research in software engineering. It is difficult and expensive to do empirical research in our field, especially the type of research that can be credibly generalized to complex, real-world projects.

I think it should be obvious that under this definition, there are no best practices in software testing.

So when someone says that we should adopt some of Testing’s best practices, should we really think that this is what they mean?

Sometimes, some individuals give every indication of actually meaning exactly this. I see them peddle “best practices” to executives, government officials, and other people who have influence or authority but not enough technical sophistication. I think those individuals, when they do this, are behaving badly. Readers who know me will probably remember situations in which I have demonstrated that I’m not shy about confronting individuals and telling them that I think they are behaving badly.

But I don’t read these statements as being about a few, specific, unprincipled people. What I think I’m seeing is a general claim that anyone who talks about best practices is either a liar or a fool (incompetent, ignorant, take your pick).

I don’t think it is productive to assume that a popular assertion is made only by liars or fools. I think it is more likely that most of the people who make the assertion mean something else, something that is not obviously incorrect.

Context-driven testers often set aside the idea that any term in testing has One True Definition. I think you have to get passed the assumption (or insistence) that any term has One True Definition or you can’t do context-driven testing. People use the same words to mean different things.

For example, many advocates of context-driven testing point out that “test case” has lots of different meanings. When someone tells you that they want you to create “test cases”, there is no value in putting your favorite meaning into their mouth. You have to ask them what they mean or you will give them the “wrong” thing.

We learned this lesson as necessary for guiding our technical practice. Why is it so hard to learn it more generally?

Rather than putting a One True Definition into the mouth of someone who says “best practice”, I think the context-driven way is to ask them what they mean by that term, and if they use an unexpected definition, to say “Oh, now I know how to interpret what you are saying” instead of saying “If you use that term with that meaning, you are telling lies.”

2. An alternative interpretation

When Joe says that in this situation, using Technique X is a best practice, Joe might mean that he has used Technique X and it worked well, that he has seen Technique X used by others and it worked well, and/or he has read reports from people he trusts that Technique X worked well. If, in Joe’s knowledge and experience, Technique X has worked better, or more reliably, than anything else he’s seen tried, Joe is likely to call Technique X “a best practice.”

At least, that’s my experience. When I’ve asked software practitioners what they mean by “best practice”, they tell me about examples that they have experienced, seen or read about. When I press practitioners for scientific evidence, they don’t have any, they typically don’t claim to have any, and they often think I’m unreasonable for expecting them to have any. (It is difficult and expensive to do well-controlled scientific research in our field, so of course they don’t have any.)

So when Joe says X is a best practice, we could interpret what Joe says as an honestly-intended assertion that X is the best thing he is aware of for this situation and he has seen enough indications that it works well that he recommends it. In my experience, that’s probably what Joe actually means.

As a specific example (because his name has come up in this discussion), my impression from talks with Rex Black is that this is what he typically means by “best practice.” He certainly has more experience with some practices than others (so some of his recommendations are better-grounded than others), but I don’t doubt that he believes that he is giving good advice.

I don’t agree with all of Rex’s recommendations. I think he favors some practices that I think are pretty terrible. But I can disagree with the man without thinking he is a liar or a fool.

And if I am willing to set aside the straw man (the version of the definition that, if true, would make Rex a liar or a fool), then I can give myself the opportunity to learn from him.

The price of prejudice

Software development is a young field. Our knowledge of the field is evolving.

Let me put this more strongly. 100 years from now, when people look back at our beliefs and generalizations, they will call them “quaint” (which is a gentle way of saying “obviously wrong”).

We all work under conditions of uncertainty. We don’t know the “truth”, so we muddle through as well as we can.

This isn’t unique to software development. Professor Billy V. Koen (a winner of ASEE’s Sterling Olmsted Award) writes about the pervasiveness of heuristics in all areas of engineering: see his books: Discussion of the Method: Conducting the Engineer’s Approach to Problem Solving (2003) and the shorter but to me more powerfully written Definition of the Engineering Method (1985). Or read his historical article, Billy V. Koen: The engineering method and the heuristic: A personal history (“This was the beginning of a 37 year quest to find one thing that was not a heuristic.”)

When we work under uncertainty, we accept and apply ideas that will later turn out to be wrong. And we reject ideas that will later turn out to be reasonable.

I have personally experienced plenty of both. For example, I advocated for exploratory testing back when it was deeply unfashionable. As to my mistakes, read The Ongoing Revolution in Software Testing for some examples.

For the field to advance, we have to be willing to learn from people we disagree with–to accept the possibility that in some ways, they might be right.

When we reject people who disagree with us as liars and fools, we close our minds and we shut down discussion. We cut ourselves off from a kind of debate, and a kind of critical thinking, that I think is essential to the personal growth of senior practitioners and to the progress of the field.

So where is the difference between context-driven testing and best practices?

In my experience, people who espouse best practices (whatever they mean by that) start from the practice when they are deciding what to do. Whether they are fans of automated GUI-level test scripts or session-based test management, they start from the position that their preferred thing is probably the solution to your problem. Then they analyze the situation and probably adapt to it, which might mean modifying the practice or picking an alternative if the first choice won’t work in your situation.

In contrast, the context-driven tester starts from the situation. What’s going on here? What do people want? What are their constraints? After they have an understanding of much of the context, they decide what to do.

Some of my colleagues would disagree with my summary (I admit that it is unflattering), but as I see it:

• They (best practice advocates) work from the solution to the problem (and perhaps try to change the problem to fit the solution)
• We (context-driven advocates) work from the problem to the solution (trying to change the solution to fit the problem).

I think this is an important difference.

But we don’t have to demonize each other over it.

Who speaks for the context-driven school?

To this point, I’ve argued that it is wrong (meaning “bad”) to censure people who advocate for “best practices.”

There is another issue: Does the context-driven school censure these people?

My answer is no. The context-driven school doesn’t censure anyone and it misrepresents the school to claim otherwise.

So, who gets to say that “the context-driven community censures… anything?”

• The founders of the school? — I think the most widely recognized founders would be Brian Marick, Bret Pettichord, James Bach and me. (Brian, James and I co-founded and co-moderated the software-testing list on egroups as the first “home” of the school. Bret, James and I wrote Lessons Learned in Software Testing.) Of these four, I think that Bret, Brian and I would flatly refuse to condemn groups of people for this view and that we would flatly refuse to call someone a liar or a fraud for believing something that we don’t believe or that we consider unsupportable.
• The elected representatives of the school? — None exist.
• The professional society that is the home of the school? — maybe the Association for Software Testing could claim this role. I strongly doubt they would issue such a condemnation. In particular, I think that at all but one of the Conferences of the Association for Software Testing (the exception was in the year James Bach chaired the conference), AST welcomed ASTQB as a corporate sponsor. Having been an executive of AST for much of that time, I can say that we would accept sponsorships from people we disagreed with, but not from people who we thought were liars and frauds. In general, AST has stood for the principle of honoring the diversity of ideas in our field even as it espouses a particular viewpoint.
• The software-testing email listserv? We started that list as a safe place for context-driven sympathizers to think together. But we did not intend it to represent the context-driven community. There are no processes in place for making decisions based on the consensus of the list. And there has been no effort to ensure that everyone who holds a context-driven view feels welcome on the list. Some people came to the list and left because they didn’t like our style. Others were kicked off. Many never joined. Of the people who are on the list, relatively few write messages to the list, partially because some of them feel intimidated. At this point, I think the list is more a collection of fans of Bach & Bolton’s Rapid Software Testing approach than of context-driven testing. Therefore, I do not think this group has the authority to damn people on behalf of “the context-driven community” either.

I don’t think anyone has the authority to censure anyone else in the name of the context-driven school.

I think it’s fair to make statements like, “I don’t think X is consistent with context-driven views” or “I think these people are liars” — but those statements put the words in the mouth of the speaker, not the mouth of the school. That’s where those words belong.