Censure people for disagreeing with us?

A few weeks ago, a colleague of mine tweeted an assertion that the context-driven testing school “censures” people who advocate for best practices.

I responded: “No one has the authority to censure people on behalf of the school” and “Surely we can disagree with people without censuring them.”

The discussion resurfaced on the “software-testing” listserv on yahoogroups, with statements like these:

  • “Context-driven is important as an antidote to the nonsense that otherwise pervades the industry. Consulting companies the world over are promoting best-practices. But there *are* no best practices. Therefore they are lying. They are promoting cynical, self-serving practices, actually.”
  • “I ‘know’ they are deliberately committing fraud for gain, as much as I can know anything in this industry.”
  • “To say there are best practices is to say something that is not true, and that any competent person knows is not true, and that any incompetent person is not qualified to be saying at all. It is indefensible, except that people who try to defend it generally say that they DIDN’T REALLY MEAN IT (that’s what Rex Black once told me) which is to say they were telling lies.”

A few people on software-testing–most notably Fiona Charles–stated their disagreement with these generalizations. But not many. Some agreed. Most were silent.

I think that calling people liars for advocating a commonly-espoused point of view is beyond unreasonable. To me, if seems like hate speech: an unfair and inaccurate characterization of many of the criticized group, likely to stir up negative emotions and to reinforce negative stereotypes.

Context-driven testing is about finding ways to do excellent testing in the actual context of the project. We adapt to the project. That includes adapting to the views and practices of the people doing that project. To do this, it is a fundamental skill for context-driven testers to be able to listen sympathetically, find common ground, and work effectively with people who have other points of view. Slamming groups of people as liars is antithetical to this. It promotes closed-mindedness, which for a context-driven tester, means ineffectiveness.

The underlying disagreement

Suppose that Joe says that in a certain situation, using Testing Technique X is a “best practice.” What does this mean?

1. The straw man

One interpretation is that Technique X is the genuinely best thing to do in this situation. That we know this to be true because we have done the enormous amount of research that would be needed to demonstrate its superiority.

Unfortunately, we don’t do this caliber of research in software engineering. It is difficult and expensive to do empirical research in our field, especially the type of research that can be credibly generalized to complex, real-world projects.

I think it should be obvious that under this definition, there are no best practices in software testing.

So when someone says that we should adopt some of Testing’s best practices, should we really think that this is what they mean?

Sometimes, some individuals give every indication of actually meaning exactly this. I see them peddle “best practices” to executives, government officials, and other people who have influence or authority but not enough technical sophistication. I think those individuals, when they do this, are behaving badly. Readers who know me will probably remember situations in which I have demonstrated that I’m not shy about confronting individuals and telling them that I think they are behaving badly.

But I don’t read these statements as being about a few, specific, unprincipled people. What I think I’m seeing is a general claim that anyone who talks about best practices is either a liar or a fool (incompetent, ignorant, take your pick).

I don’t think it is productive to assume that a popular assertion is made only by liars or fools. I think it is more likely that most of the people who make the assertion mean something else, something that is not obviously incorrect.

Context-driven testers often set aside the idea that any term in testing has One True Definition. I think you have to get passed the assumption (or insistence) that any term has One True Definition or you can’t do context-driven testing. People use the same words to mean different things.

For example, many advocates of context-driven testing point out that “test case” has lots of different meanings. When someone tells you that they want you to create “test cases”, there is no value in putting your favorite meaning into their mouth. You have to ask them what they mean or you will give them the “wrong” thing.

We learned this lesson as necessary for guiding our technical practice. Why is it so hard to learn it more generally?

Rather than putting a One True Definition into the mouth of someone who says “best practice”, I think the context-driven way is to ask them what they mean by that term, and if they use an unexpected definition, to say “Oh, now I know how to interpret what you are saying” instead of saying “If you use that term with that meaning, you are telling lies.”

2. An alternative interpretation

When Joe says that in this situation, using Technique X is a best practice, Joe might mean that he has used Technique X and it worked well, that he has seen Technique X used by others and it worked well, and/or he has read reports from people he trusts that Technique X worked well. If, in Joe’s knowledge and experience, Technique X has worked better, or more reliably, than anything else he’s seen tried, Joe is likely to call Technique X “a best practice.”

At least, that’s my experience. When I’ve asked software practitioners what they mean by “best practice”, they tell me about examples that they have experienced, seen or read about. When I press practitioners for scientific evidence, they don’t have any, they typically don’t claim to have any, and they often think I’m unreasonable for expecting them to have any. (It is difficult and expensive to do well-controlled scientific research in our field, so of course they don’t have any.)

So when Joe says X is a best practice, we could interpret what Joe says as an honestly-intended assertion that X is the best thing he is aware of for this situation and he has seen enough indications that it works well that he recommends it. In my experience, that’s probably what Joe actually means.

As a specific example (because his name has come up in this discussion), my impression from talks with Rex Black is that this is what he typically means by “best practice.” He certainly has more experience with some practices than others (so some of his recommendations are better-grounded than others), but I don’t doubt that he believes that he is giving good advice.

I don’t agree with all of Rex’s recommendations. I think he favors some practices that I think are pretty terrible. But I can disagree with the man without thinking he is a liar or a fool.

And if I am willing to set aside the straw man (the version of the definition that, if true, would make Rex a liar or a fool), then I can give myself the opportunity to learn from him.

The price of prejudice

Software development is a young field. Our knowledge of the field is evolving.

Let me put this more strongly. 100 years from now, when people look back at our beliefs and generalizations, they will call them “quaint” (which is a gentle way of saying “obviously wrong”).

We all work under conditions of uncertainty. We don’t know the “truth”, so we muddle through as well as we can.

This isn’t unique to software development. Professor Billy V. Koen (a winner of ASEE’s Sterling Olmsted Award) writes about the pervasiveness of heuristics in all areas of engineering: see his books: Discussion of the Method: Conducting the Engineer’s Approach to Problem Solving (2003) and the shorter but to me more powerfully written Definition of the Engineering Method (1985). Or read his historical article, Billy V. Koen: The engineering method and the heuristic: A personal history (“This was the beginning of a 37 year quest to find one thing that was not a heuristic.”)

When we work under uncertainty, we accept and apply ideas that will later turn out to be wrong. And we reject ideas that will later turn out to be reasonable.

I have personally experienced plenty of both. For example, I advocated for exploratory testing back when it was deeply unfashionable. As to my mistakes, read The Ongoing Revolution in Software Testing for some examples.

For the field to advance, we have to be willing to learn from people we disagree with–to accept the possibility that in some ways, they might be right.

When we reject people who disagree with us as liars and fools, we close our minds and we shut down discussion. We cut ourselves off from a kind of debate, and a kind of critical thinking, that I think is essential to the personal growth of senior practitioners and to the progress of the field.

So where is the difference between context-driven testing and best practices?

In my experience, people who espouse best practices (whatever they mean by that) start from the practice when they are deciding what to do. Whether they are fans of automated GUI-level test scripts or session-based test management, they start from the position that their preferred thing is probably the solution to your problem. Then they analyze the situation and probably adapt to it, which might mean modifying the practice or picking an alternative if the first choice won’t work in your situation.

In contrast, the context-driven tester starts from the situation. What’s going on here? What do people want? What are their constraints? After they have an understanding of much of the context, they decide what to do.

Some of my colleagues would disagree with my summary (I admit that it is unflattering), but as I see it:

  • They (best practice advocates) work from the solution to the problem (and perhaps try to change the problem to fit the solution)
  • We (context-driven advocates) work from the problem to the solution (trying to change the solution to fit the problem).

I think this is an important difference.

But we don’t have to demonize each other over it.

Who speaks for the context-driven school?

To this point, I’ve argued that it is wrong (meaning “bad”) to censure people who advocate for “best practices.”

There is another issue: Does the context-driven school censure these people?

My answer is no. The context-driven school doesn’t censure anyone and it misrepresents the school to claim otherwise.

So, who gets to say that “the context-driven community censures… anything?”

  • The founders of the school? — I think the most widely recognized founders would be Brian Marick, Bret Pettichord, James Bach and me. (Brian, James and I co-founded and co-moderated the software-testing list on egroups as the first “home” of the school. Bret, James and I wrote Lessons Learned in Software Testing.) Of these four, I think that Bret, Brian and I would flatly refuse to condemn groups of people for this view and that we would flatly refuse to call someone a liar or a fraud for believing something that we don’t believe or that we consider unsupportable.
  • The elected representatives of the school? — None exist.
  • The professional society that is the home of the school? — maybe the Association for Software Testing could claim this role. I strongly doubt they would issue such a condemnation. In particular, I think that at all but one of the Conferences of the Association for Software Testing (the exception was in the year James Bach chaired the conference), AST welcomed ASTQB as a corporate sponsor. Having been an executive of AST for much of that time, I can say that we would accept sponsorships from people we disagreed with, but not from people who we thought were liars and frauds. In general, AST has stood for the principle of honoring the diversity of ideas in our field even as it espouses a particular viewpoint.
  • The software-testing email listserv? We started that list as a safe place for context-driven sympathizers to think together. But we did not intend it to represent the context-driven community. There are no processes in place for making decisions based on the consensus of the list. And there has been no effort to ensure that everyone who holds a context-driven view feels welcome on the list. Some people came to the list and left because they didn’t like our style. Others were kicked off. Many never joined. Of the people who are on the list, relatively few write messages to the list, partially because some of them feel intimidated. At this point, I think the list is more a collection of fans of Bach & Bolton’s Rapid Software Testing approach than of context-driven testing. Therefore, I do not think this group has the authority to damn people on behalf of “the context-driven community” either.

I don’t think anyone has the authority to censure anyone else in the name of the context-driven school.

I think it’s fair to make statements like, “I don’t think X is consistent with context-driven views” or “I think these people are liars” — but those statements put the words in the mouth of the speaker, not the mouth of the school. That’s where those words belong.

6 thoughts on “Censure people for disagreeing with us?

  1. One thing I love about the context-driven school is a certain adherence to the rules of scientific debate. One must defend ones views, and it’s almost a requirement to question bad practice whenever one sees it, but I think when someone starts making pronouncements on behalf of the context-driven school they are helping cement it into place as a dogmatic temple of nay-sayers, rather than the mixture of opinion and disagreements it is. “We” disagree with each other, and therefore “we” do not have a single agreed voice with which to voice vague attacks on anyone or anything.

    Saying “the context-driven school censures…” is really saying “members of the context-driven school censure…” which is just as false a logic as saying “members of the X-school lie”.

    Saying that, aren’t statements like “Context-driven is important as an antidote to the nonsense..” a voiced opinion on context-driven testing from someone who happens to be a context-driven tester? Perhaps it is antithetical to promoting common ground, but what if common ground is not the goal? Perhaps the goal is finding what is true, or challenging what is false.

    As for calling people liars perhaps we should be doing what we do best – asking questions of those people. Where’s the evidence that they are liars? What leads to that conclusion? How did you reach the idea? Apply the same ideas you espouse to conflicts and common ground with other schools to understanding the dogmatic of our “own” crowd.

    I am worried somewhat, not for the expression of opinion (that we can question, pick apart and challenge), but the censorship of ideas. Whenever we seek to silence someone with a wild and crazy opinion we deny ourselves the opportunity to hear what they have to say, which may contain a grain of truth, probably took that person some effort to come up with, and gives us pause to consider why we know what we think we know – how do I know something is the case, is it just because I’ve been told it?

    I think it’s as important to allow people to make generalised statements (and have them challenged) as it is to promote good ideas, and a change in the language (such as “best practices” and the context-driven understanding of it).

    That niggle out of the way I have to say that I do agree with what you say. It’s a bit hypocritical to make statements from a single perspective and call yourself a “context-driven” tester at the same time, as it’s fairly obvious that that person doesn’t understand the power of context. It’s a school of thought on testing, not a church. You could read “There are no best practices, there are only good practices in context” as “We don’t have a Bible, we do what we can with what we have”

    Thanks for the good read!

  2. Although I would not call myself a Context Driven Tester (at least not yet), I have met many people who are, and as a community they do inspire me. There are some differences, but throughout them all, there is a passionate drive to deliver testing which means their customers (true) project needs.

    For companies who open their minds, Context Driven Testers can deliver true value – they can challenge companies stereotypes about what testers do and what they deliver (which is metrics and test scripts right???).

    Make no mistake, the tag “Context Driven Tester” is becoming in some circles a valuable brand.

    So what’s to stop the same people who claim to their customers today “we deliver best practices” waking up one morning and saying “well I’d like a piece of this Context Driven Tester business … I’m now a Context Driven Tester!”. There’s no certification or piece of paper, you just start putting it on your CV.

    So all of a sudden there’s someone going around the Software Community saying “I’m a Context Driven Tester … and I bring my best practices and metrics and software scripts over exploratory testing”.

    Someone like that undermines the principles, and their failures are very public. We’ve already faced that terrible issue in the Agile community, where people have started with Agile principles, and Waterfalled them until they broke (see “We tried baseball and it didn’t work”).

    Shouldn’t someone who claims to be a Context Driven Tester (very publically) but basically undermines not just the principles but the community that it represents be challenged. I would claim first and foremost they should need to be educated (again you see this in Agile where someones ideas shot down as “that’s not Agile”).

    But if they really are doing damage, then maybe they need to be not just censured but decried as a fake?

    The IT Industry is already too filled with PT Barnums who promise tools, solutions, practices which just fail to deliver. It falls to testers with passion to pick up the piece, dispel the myths and repair the damage.

  3. The Cynefin framework would suggest that appreciation of context determines when best practices should be developed and applied, but that isn’t really what this post was about.

    The post calls for civil discourse, which seems to be rare in discussion forae these days and visibly in current US politics (http://www.kofc.org/en/news/releases/detail/poll_civilityinamerica_20120726.html). The post concludes with a call to personal accounatbility rather than false ad populum or ad vericundiam calls. I read ad hominems into questioning the motives of any person.

    We dare ferociously challenge any idea without calling into question the character of those who espouse it. If we choose instead to attack character, we have missed the target and may lose the fight and condemn ourselves as a boxer disqualifed for hitting below the belt.

    Since the post is really about professional conduct in public discussion…

    If we are professionals, what do we profess? If we are empiricists, where are our data? And if we consider our ways better, we dare not believe them best — someone will come up with a better idea, perhaps even we if we remain open to challenging ourselves.

  4. Cem,
    About the schools of testing and about using these to divide/slot/label people, I wrote something recently. Having been a student of philosophy and you having read about Hinduism – here is the link. http://shishya-eternalstudent.blogspot.in/2012/10/religion-india-and-testing.html


    Thank you for a thoughtful comment. I enjoyed reading your post.

    I think you know that my grandmother was a Buddhist and that, as I matured, I blended many ideas from India with my primarily Jewish upbringing. This was a foundation for much of my skepticism about rigid models that assumed a specification would (or could) give the one “true” description of a complex system–how much of that “truth” is an illusion or an unrecognized ambiguity? The question, “what if this is not quite what it seems?” is the natural outgrowth of this, and the basis (for me) of exploratory testing.

    I think you are saying (and if you are, I agree), that it is the rigidity–the insistence on the One Correct Way of analyzing things that makes every other way Wrong–that is the fundamental error in how we can see and interact with the world.

    — Cem

  5. I chose not to engage in those dogmatic discussions. I once had a job interview where the term context-driven led one of the devs to do some googling. I had to defend myself for affiliating as he’d found some right contentious and dogmatic stuff and wondered if I were some kind of extremist for including that term in my resume. It’s no longer in my resume, FWIW.

    I thought this comment in that thread spoke volumes to me:

    “We are not going to debate whether or not there are best testing practices on this forum.

    If you think there are, you will have to go debate that somewhere else, because we will shut you down.”

    I choose to read that as “Ideas that don’t agree with ours are not welcome here and we will not welcome those who express those beliefs.” I think it’s the height of arrogance and antithetical to my personal ethic of challenging my beliefs. I think a willingness to expose ignorance publicly and assess one’s core values is important. Exchanges like those remind me of a great video of Neil De Grasse Tyson “rebuking” Richard Dawkins for being dogmatic : http://youtu.be/-_2xGIwQfik

    Also I encourage folks to read Chris Argyris’ work on single vs double-loop learning and when someone who professes to be a leader says things like “We will shut you down” ask yourself if they’re exhibiting model 1 or model 2 behavior before you decide how to respond.

    Anyway, great post Cem. I don’t belong to clubs in general partly because I believe it is the nature of groups of people to eventually devolve into divisiveness and however inadvertently start talking about censuring folks outside that club. I applaud your position on this subject and remain a fan.

    Adam Yuret

  6. Thanks Cem for your words. Yes, I am saying exactly that (“that it is the rigidity–the insistence on the One Correct Way of analyzing things that makes every other way Wrong–that is the fundamental error in how we can see and interact with the world”)
    and also that ideas can be categorized but human beings should not be esp. if the categorization leads to censure, ridicule, superiority/inferiority.

Comments are closed.