I don’t usually let myself get dragged into the TwitterStorms orchestrated by the Rabid Software Testing crowd. But I decided to make an exception for this one because it has sucked some good people into endorsing what I think is a very bad petition.

In essence, the petition asks ISTQB to open its quality-control records. ISTQB is involved in training and certifying software testers, so I will write this to you (the reader) as if you are a software tester. The basic question is whether ISTQB has seen any problems in their exams and what they have done about it. (Underneath that are some more specifically-pointed details, which I’ll come to later.)

Let’s Start On This From The Basic Principle

In many countries (such as the United States), companies have a right to study the quality of their goods or services in private. That’s what I understand that ISTQB has been doing.

By the way, that’s what companies do when they test their software. Those companies (ISTQB, and software companies) have a right to hire employees and consultants and to require those people to keep what they’ve learned private. When you test your company’s or client’s software, it is normally a condition of your job or your consultancy that you keep your results private.

This petition asks ISTQB to do what (for the most part) the companies you work for would not do, and would not dream of letting you do with their quality-control data.

When I studied law, I learned that the public policy of the United States (and many other countries) favors investigative privilege. That is, we encourage companies to conduct aggressive, detailed internal investigations, to find their own problems and to improve their products, their services and their advertising based on what they learn from the investigation. We want them to investigate because we want the improvement.

These types of studies expose the companies to risk–people who get their hands on the investigative data, and don’t like those company that investigates itself, can use any report of any problem, whether they understand the problem or not, whether the problem is actually major or not, whether the company has dealt with the problem in a reasonable way or not–to attack the company. Therefore, to encourage companies to do the studies, society grants them privacy–we treat the studies as trade secret–because the companies won’t do serious studies without privacy.

When I practiced commercial law, my most significant project involved drafting legislation about the law of software quality. My most significant contribution was a very narrow disclosure rule that would preserve software vendors’ incentives to do internal investigations (i.e. test their own software harshly) while holding them accountable for significant defects that they knew about but did not make public. It took 6 years to craft the language for this. It took 6 years because the idea of almost-forced disclosure of (some) test results triggered allergic reactions among lawyers who normally defended companies, among lawyers who normally attacked companies, and among judges. Finding a way to balance the interests here was enormously difficult because the support for allowing companies to investigate their products and services in private runs so very deep. And, in my view, it should.

And if you want to be able to do significant work as testers, you should support that privilege too.

And that means you should show some respect for that privilege, even when the company asserting it is a company that you (or some people who impress you by screaming at others in public) don’t like.

Now, we have a petition from someone who frequently attacks ISTQB, asking them to open their private quality-control records associated with their certification exams. Will I sign it?

No, Keith. I will not sign that petition.

A Little More Background

Back in 2010, I saw some materials that allegedly summarized unauthorized disclosures from an ISTQB meeting. I suspect that this (2010) material underlies the specificity of the questions in this (2013) petition to ISTQB.

The materials were not authenticated and there might not be any truth in them at all. The gist of the materials was that ISTQB had commissioned a psychometric study of one or more of their tests, that a “test reliability coefficient” was a bit low, and that ISTQB was planning to use this information to improve the quality of their exams.

I believe that this is the type of research that we, as a society (and as a profession of testers), want to see done. Look for problems. When you find problems, raise them with other decision-makers in the organization and look for ways to improve them.

To keep this going, society respects these studies as trade secret–grants them privacy. There is no expectation, zero, none, that they will tell us about the studies or what they did to address them.

(There are some exceptions to that rule. I will note them later, to suggest that they are probably not relevant here.)

Certain people encouraged me to get involved in attacking ISTQB with these allegations. I refused.

One thing I said to those people, back in 2010, was:

“I’m not a huge fan of attacking competitors. I fall into it, but I think it is a distraction. I think it adds to the personality circus of the field and detracts from the content of the field.”

Another of the things I said was:

“As to the ethics, I am slightly outraged by the fact that ISTQB apparently will not make this public. But I am also slightly outraged that this was leaked, if it was. You’re being offered the opportunity to play the role of one hostile competitor viciously attacking another competitor. Your attack will not look ethical.”

A little later, I looked more carefully at this “test reliability coefficient”. I am not an expert in this area of statistics, but I know a little bit.

• My impression is that the theory underlying the meaning and application of the statistic is not fully developed.
• My impression is that the cutoff that separates “good” values from not-good-enough values is arbitrary.
• My impression is that something like this is probably useful for internal studies, because it can raise flags. A not-excellent number can motivate people to improve, even if that number is not necessarily very bad, and even if the other aspects of the meaning of the number are a little uncertain.

I’ve dealt with lots of metrics like this in my career. They are good enough for private, informal, self-evaluation or evaluation by a friendly coach. But the idea of attacking a company based on not-terrible-but-not-great values of these statistics, well, I don’t think that’s a valid use of this statistic. Not without a whole lot of other converging evidence.

You may have seen me quoted about these types of metrics once or twice. The quote goes:

“Metrics that are not valid are dangerous.”

So, once I studied the statistic a little, I was no longer even a little bit outraged that ISTQB wasn’t making the data public. It was not only their privilege to keep it private. Keeping it private was also, in my view, reasonable.

I think that using metrics like this to publicly attack people (or companies) is unethical. Even if they are people or companies that I disagree with, or don’t like.

So, Keith, No, Keith. I will not sign that petition.

The Big Public Attack on ISTQB over this didn’t happen in 2010 and I expected never to hear of these allegations again.

Should We Ever Force Companies To Disclose Private Data?

Sometimes, society forces companies to disclose their private data.

For example, if a company commits crimes (you know, like rigging LIBOR, or engaging in tax evasion, or laundering money), its victims (and prosecutors representing society as a whole) can demand to see relevant records and the courts will enforce the demand.

Suppose that ISTQB made specific comments about the reliability of their tests, and they were in possession of data that indicated that those specific comments were false. That would be fraud. In that case, it should be possible to obtain ISTQB’s internal data.

I’m not a big fan of ISTQB’s syllabus or exams, or of their marketing. My attitudes are not as extreme as some other people’s. Some twits seem to believe that ISTQB’s materials are worthless, that everyone finds them worthless, and that no reasonable person thinks the exams or the certifications are worth anything.

• I have been told by some reasonable people that they believed they learned a great deal from ISTQB courses.
• I have been told by some reasonably bright people that they found an ISTQB exam challenging.
• I have been told by some teachers that they see value in the material.
• I have been told by some hiring managers who have testing experience that they believe that this should be a positive hiring factor.

That’s not exactly my evaluation, but I have no reason to doubt the integrity of these other people, or doubt their experience.

But because I’m skeptical of ISTQB, every now and again, I look at their marketing materials. I haven’t looked at everything. I haven’t even looked at most of the materials. But in what I have seen, even though I don’t agree with the views being expressed or the conclusions they would have me draw:

• I have not seen evidence of fraud
• I have not seen claims that could be refuted by mediocre results in test reliability scores.

So. Keith. No, Keith. Absolutely not, Keith. I will not sign that petition.

I encourage you to withdraw it.