- Testers provide information to our clients (stakeholders) about the product, about how we tested it, and about what we found.
- Our clients get to decide what information they want. We don’t get to decide that for them.
- Testers provide services to software projects. We don’t run the projects. We don’t control those projects’ contexts.
In context-driven testing, we respect the fact that contexts differ.
What Does “contexts differ” Really Mean?
I think it means that in different contexts, the people who are our clients:
- are going to want different types of information
- are going to want us to prioritize our work differently
- are going to want us to test differently, to mitigate different risks, and to report our results in different ways.
Contexts don’t just differ for the testers. They differ for the project managers too. The project managers have to report to other people who want whatever information they want.
We don’t manage the project managers. We don’t decide what information they have to give to the people they report to.
Sometimes, Our Clients Want Metrics
Sometimes, a client will ask how many test cases the testers have run:
- I don’t think this is a very useful number. It can be misleading. And if I organize my testing with this number in mind, I might do worse testing.
- So if a client asks me for this number, I might have a discussion with her or him about why s/he thinks s/he needs this statistic and whether s/he could be happy with something else.
- But if my client says, “No, really, I need that number“, I say, OK and give the number.
Sometimes a client will ask about defect removal efficiency:
- I think this is a poor excuse for a metric. I have a nice rant about it when I teach my graduate course in software metrics. Bad metric. BAD!
- If a client asks for it, I am likely to ask, Are you sure? If they’re willing to listen, I explain my concerns.
But defect removal efficiency (DRE) is a fairly popular metric. It’s in lots of textbooks. People talk about it at conferences. So no matter what I say about it, my client might still want that number. Maybe my client’s boss wants it. Maybe my client’s customer wants it. Maybe my client’s regulator wants it. This is my client’s management context. I don’t think I’m entitled to know all the details of my client’s working situation, so maybe my client will explain why s/he needs this number and maybe s/he won’t.
So if the client says, “No, really, I need the DRE“, I accept that statement as a description of my client’s situation and I say, OK and give the number.
One more example: ratio of passing to failing tests. Michael Bolton presents several reasons for disliking this metric and I generally agree with them. In particular, I don’t know what the ratio measures (it has no obvious construct validity). And if the goal is to make the number big, there are lots of ways to achieve this that yield weak testing (see Austin on measurement dysfunction, for discussion of this type of problem.)
But Michael takes it a step further and says that using this metric, or providing it to a client is unethical.
If you give this metric to someone (after they ask, and you say it’s not very good, and they say, really-I-want-it):
- Are you lying?
- Are you taking something that doesn’t belong to you?
- Are you oppressing someone? Intimidating them?
- Are you hurting someone?
- Are you cheating anyone?
- Are you pretending to skills or knowledge that you don’t have?
- Are you helping someone else lie, cheat, steal, intimidate, or cause harm?
I used to associate shrill accusations of unethicalness with conservatives who were losing control of the hearts and minds of the software development community and didn’t like it, or who were pushing a phony image of community consensus as part of their campaigns to get big contracts, especially big government contracts, or who were using the accusation of unethical as a way of shutting down discussion of whether an idea (unethical!) was any good or not.
Maybe you’ve met some of these people. They said things like:
- It is unethical to write code if you don’t have formal, written requirements
- It is unethical to test a program if you don’t have written specifications
- It is unethical to do exploratory testing
- It is unethical to manage software projects without formal measurement programs
- It is unethical to count lines of code instead of using function points
- It is unethical to count function points instead of lines of code
- It is unethical to not adopt best practices
- It is unethical to write or design code if you don’t have the right degree or certificate
- It should be unethical to write code if you don’t have a license
It seemed to me that some of the people (but not all of the people) who said these things were trying to prop up a losing point of view with fear, uncertainty, doubt — they were using demagoguery as their marketing technique. That I saw as unethical.
Much of my contribution to the social infrastructure of software testing was a conscious rebellion against a closed old boys network that defended itself with dogma and attacked non-conformers as unethical.
wrong versus Wrong
So what’s with this “Using a crummy metric is unethical” ?
Over the past couple of years, I’ve seen a resurgence of ethics-rhetoric. A new set of people have a new set of bad things to condemn:
- Now it seems to be unethical to have a certification in software testing that someone doesn’t like
- Now it seems to be unethical to follow a heavyweight (heavily documented, scripted) style of testing
- Now it seems to be unethical to give a client some data that the client asks for, like a ratio of passing tests to failing ones.
I don’t think these are usually good ideas. In fact, most of the time, I think they’re wrong.
I’m not a moral relativist. I think there is evil in the world and I sometimes protest loudly against it. But I think it is essential to differentiate between:
- someone is wrong (mistaken)
- someone is wrong (attempting something that won’t work), and
- someone is Wrong (unethical).
Let me illustrate the difference. Michael Bolton is a friend of mine. I have a lot of respect for him as a person, including as an ethical being. His blog post is a convenient example of something I think is a broader problem, but please read my comments on his article as an assertion that I think Michael is wrong (not Wrong).
To the extent that we lose track of the difference between wrong and Wrong, I think we damage our ability to treat people who disagree with us with respect. I think we damage our ability to communicate about our professional differences. I think we damage our ability to learn, because the people we most agree with probably have fewer new things to teach us than the people who see the world a little differently.
The difference between wrong and Wrong is especially important for testers who want to think of ourselves (or market ourselves) as context-driven.
Because we understand that what is wrong in some contexts is right in some others.
I appreciate your effort to clarify the question of the role of ethics in software testing. I would like to add some thoughts to that:
1. If a stakeholder asks for e.g. “how many test cases were run by X” then according to your definition this would be classified as either “mistaken” or “attempting something that won’t work”. (please correct me if my inference is incorrect)
Now, if based on the delivery of that number a contract of a tester is no longer renewed or the tester is fired, then in my view answering that question becomes a highly ethical matter.
2. I agree that we should be careful in using ethics as a weapon. It just loses its strength if used carelessly. Maybe we should focus more on the concept of “craftsmanship”.
Reporting a “no. of test cases executed by x” just violates my sense of testing craftsmanship. I think a cabinet maker would simply refuse if a customer asked for something that would make a table unstable and unfit for use.
Can you elaborate on what you see here as the ethical issue? I’d like to understand your meaning a little more precisely before responding.
My understanding of making a good ethical decision is to decide on a course of action that I as an actor can evaluate as right according to my model of what is right.
My model says: A tester being fired as a result of misleading data I provided to somebody is wrong.
Therefore, confronted with the question if I should provide misleading data becomes an ethical question to me.
So the first question is whether pass/fail data are inherently misleading. Can pass/fail data be used in a way which is not misleading?
One of the critical problems of metrics is that we talk about specific statistics (such as a pass/fail ratio) without talking about what the statistic will be used to signify. It is the hypothesized relationship between the statistic and some underlying attribute that can be misleading, not the statistic itself. (The statistic on its own has no meaning and therefore cannot be misleading. We give it meaning–this is the problem of constructs and construct validity.)
In this case, you posit that the statistic is interpreted as a key performance measure of some tester. I agree with you that it is probably not a good human performance measure. (I can only say “probably” because I don’t know what it is _supposed_ to measure.) So in this case, a manager will use a metric that is invalid for that purpose to do harm to someone. Obviously, that would be unethical. And if I knew that this was the intent of the manager, then I would probably refuse to supply the data.
But we have now added back all sorts of context that helps us understand the situationally-specific ethical problem.
What of the manager who says to the tester, “I want pass/fail ratios” and the tester says, “They are crummy metrics” and the manager says, “We have recorded pass/fail ratios since the days of dinosaurs tapping on keyboards with their tails. I want the ratios for my records.” This manager might ignore the data, use the data in a way that might actually make some sense, misuse the data in ways that oppress no one, or misuse the data in ways that harm others. This tester does not know. Should the tester refuse to provide the data?
If pass/fail metrics are truly worst practices — any practice that is inherently unethical would seem to fit that bill — then the tester should refuse to provide these metrics in all contexts. And that is certainly what I think is being advocated by people who slam metrics like this (the metric itself, not just a specific use of it) as UNETHICAL.
If the distinction is hard to understand, then consider bananas.
If you work at a grocery store, people might sometimes ask you for bananas. If someone asks for a banana, and they are willing to pay for it, should you refuse?
Many managers like bananas. Think of all those managers of young children who say, “quit playing with your bananas and eat them!”
But bananas can be used for very bad purposes. They can be used to embarrass, to injure, to rape, to kill. Maybe we should declare bananas unethical.
Or maybe we should say that if someone asks a grocer for a banana, the normal rule is that the grocer should say “OK” and give the banana. But if the grocer knows that this person will use the banana in an evil way, in that case, the grocer should say “No. No bananas for you.”
But this is no longer a context-free, generalized condemnation of bananas as unethical.
Nor, I argue, is it appropriate to stage a context-free, generalized condemnation of ratio statistics as unethical.
And thus even a well thought out position, if positioned as context-free, can become context-imperial?
Testers often face demands that make little sense when it comes to testing well, be it a restrictively enforced standard, a request for a meaningless metrics, or a process that says to do the rain dance when a new build arrives.
These all form part of our context, and in many cases testers have neither the abillity to make it otherwise, nor the right to demand that it be so.
When faced with these situtions, I come right back to the words on this site: “Testers get what they get, and skilled context-driven testers must know how to cope with what comes their way. Of course, we can and should explain tradeoffs to people, make it clear what makes us more efficient and more effective, but ultimately, we see testing as a service to stakeholders who make the broader project management decisions.”
At the end of the day, we all have a choice: work within our context or quit so as to find a context that is more ameniable to our sensitivities.
Pingback: Five Blogs – 7 March 2012 « 5blogs
Very interesting read as always.
Pingback: Context Driven Testing » Metrics, Ethics, & Context-Driven Testing (Part 2)