I don’t usually let myself get dragged into the TwitterStorms orchestrated by the Rabid Software Testing crowd. But I decided to make an exception for this one because it has sucked some good people into endorsing what I think is a very bad petition.
In essence, the petition asks ISTQB to open its quality-control records. ISTQB is involved in training and certifying software testers, so I will write this to you (the reader) as if you are a software tester. The basic question is whether ISTQB has seen any problems in their exams and what they have done about it. (Underneath that are some more specifically-pointed details, which I’ll come to later.)
Let’s Start On This From The Basic Principle
In many countries (such as the United States), companies have a right to study the quality of their goods or services in private. That’s what I understand that ISTQB has been doing.
By the way, that’s what companies do when they test their software. Those companies (ISTQB, and software companies) have a right to hire employees and consultants and to require those people to keep what they’ve learned private. When you test your company’s or client’s software, it is normally a condition of your job or your consultancy that you keep your results private.
This petition asks ISTQB to do what (for the most part) the companies you work for would not do, and would not dream of letting you do with their quality-control data.
When I studied law, I learned that the public policy of the United States (and many other countries) favors investigative privilege. That is, we encourage companies to conduct aggressive, detailed internal investigations, to find their own problems and to improve their products, their services and their advertising based on what they learn from the investigation. We want them to investigate because we want the improvement.
These types of studies expose the companies to risk–people who get their hands on the investigative data, and don’t like those company that investigates itself, can use any report of any problem, whether they understand the problem or not, whether the problem is actually major or not, whether the company has dealt with the problem in a reasonable way or not–to attack the company. Therefore, to encourage companies to do the studies, society grants them privacy–we treat the studies as trade secret–because the companies won’t do serious studies without privacy.
When I practiced commercial law, my most significant project involved drafting legislation about the law of software quality. My most significant contribution was a very narrow disclosure rule that would preserve software vendors’ incentives to do internal investigations (i.e. test their own software harshly) while holding them accountable for significant defects that they knew about but did not make public. It took 6 years to craft the language for this. It took 6 years because the idea of almost-forced disclosure of (some) test results triggered allergic reactions among lawyers who normally defended companies, among lawyers who normally attacked companies, and among judges. Finding a way to balance the interests here was enormously difficult because the support for allowing companies to investigate their products and services in private runs so very deep. And, in my view, it should.
And if you want to be able to do significant work as testers, you should support that privilege too.
And that means you should show some respect for that privilege, even when the company asserting it is a company that you (or some people who impress you by screaming at others in public) don’t like.
Now, we have a petition from someone who frequently attacks ISTQB, asking them to open their private quality-control records associated with their certification exams. Will I sign it?
No, Keith. I will not sign that petition.
A Little More Background
Back in 2010, I saw some materials that allegedly summarized unauthorized disclosures from an ISTQB meeting. I suspect that this (2010) material underlies the specificity of the questions in this (2013) petition to ISTQB.
The materials were not authenticated and there might not be any truth in them at all. The gist of the materials was that ISTQB had commissioned a psychometric study of one or more of their tests, that a “test reliability coefficient” was a bit low, and that ISTQB was planning to use this information to improve the quality of their exams.
I believe that this is the type of research that we, as a society (and as a profession of testers), want to see done. Look for problems. When you find problems, raise them with other decision-makers in the organization and look for ways to improve them.
To keep this going, society respects these studies as trade secret–grants them privacy. There is no expectation, zero, none, that they will tell us about the studies or what they did to address them.
(There are some exceptions to that rule. I will note them later, to suggest that they are probably not relevant here.)
Certain people encouraged me to get involved in attacking ISTQB with these allegations. I refused.
One thing I said to those people, back in 2010, was:
“I’m not a huge fan of attacking competitors. I fall into it, but I think it is a distraction. I think it adds to the personality circus of the field and detracts from the content of the field.”
Another of the things I said was:
“As to the ethics, I am slightly outraged by the fact that ISTQB apparently will not make this public. But I am also slightly outraged that this was leaked, if it was. You’re being offered the opportunity to play the role of one hostile competitor viciously attacking another competitor. Your attack will not look ethical.”
A little later, I looked more carefully at this “test reliability coefficient”. I am not an expert in this area of statistics, but I know a little bit.
- My impression is that the theory underlying the meaning and application of the statistic is not fully developed.
- My impression is that the cutoff that separates “good” values from not-good-enough values is arbitrary.
- My impression is that something like this is probably useful for internal studies, because it can raise flags. A not-excellent number can motivate people to improve, even if that number is not necessarily very bad, and even if the other aspects of the meaning of the number are a little uncertain.
I’ve dealt with lots of metrics like this in my career. They are good enough for private, informal, self-evaluation or evaluation by a friendly coach. But the idea of attacking a company based on not-terrible-but-not-great values of these statistics, well, I don’t think that’s a valid use of this statistic. Not without a whole lot of other converging evidence.
You may have seen me quoted about these types of metrics once or twice. The quote goes:
“Metrics that are not valid are dangerous.”
So, once I studied the statistic a little, I was no longer even a little bit outraged that ISTQB wasn’t making the data public. It was not only their privilege to keep it private. Keeping it private was also, in my view, reasonable.
I think that using metrics like this to publicly attack people (or companies) is unethical. Even if they are people or companies that I disagree with, or don’t like.
So, Keith, No, Keith. I will not sign that petition.
The Big Public Attack on ISTQB over this didn’t happen in 2010 and I expected never to hear of these allegations again.
Should We Ever Force Companies To Disclose Private Data?
Sometimes, society forces companies to disclose their private data.
For example, if a company commits crimes (you know, like rigging LIBOR, or engaging in tax evasion, or laundering money), its victims (and prosecutors representing society as a whole) can demand to see relevant records and the courts will enforce the demand.
Suppose that ISTQB made specific comments about the reliability of their tests, and they were in possession of data that indicated that those specific comments were false. That would be fraud. In that case, it should be possible to obtain ISTQB’s internal data.
I’m not a big fan of ISTQB’s syllabus or exams, or of their marketing. My attitudes are not as extreme as some other people’s. Some twits seem to believe that ISTQB’s materials are worthless, that everyone finds them worthless, and that no reasonable person thinks the exams or the certifications are worth anything.
- I have been told by some reasonable people that they believed they learned a great deal from ISTQB courses.
- I have been told by some reasonably bright people that they found an ISTQB exam challenging.
- I have been told by some teachers that they see value in the material.
- I have been told by some hiring managers who have testing experience that they believe that this should be a positive hiring factor.
That’s not exactly my evaluation, but I have no reason to doubt the integrity of these other people, or doubt their experience.
But because I’m skeptical of ISTQB, every now and again, I look at their marketing materials. I haven’t looked at everything. I haven’t even looked at most of the materials. But in what I have seen, even though I don’t agree with the views being expressed or the conclusions they would have me draw:
- I have not seen evidence of fraud
- I have not seen claims that could be refuted by mediocre results in test reliability scores.
So. Keith. No, Keith. Absolutely not, Keith. I will not sign that petition.
I encourage you to withdraw it.
While I remain unimpressed by the bluster of @RBCS, I am inclined to agree with you, sir: no good purpose will be served by this petition effort.
Hi Cem,
Interesting post. I have to admit I do not know a lot about law so I will immediately believe you when you say it is legal not to disclose private data. But is it ethical to do so? If an organisation makes claims openly, it sounds only reasonable to me if people ask for clarification and evidence to back those claims up.
When I read these documents (http://kaner.com/pdfs/ocp1.pdf and http://kaner.com/pdfs/ocp2.pdf) on your website it seems to me that you have changed your mind. Am I right?
In these documents you say: “Software testers are professional skeptics. To require them to adopt a compliance mentality, in which they set aside issues of ambiguity, oversimplification, unstated assumptions or controversial conclusions in order to provide the answer expected by an examiner is to demand conduct so far removed from what testers should do as to be invalid on its face.” and “The emphasis on openness makes possible a more revealing and less prescriptive use of certification exams.”
These statements seem not consistent with what you say in this post. What made you change your mind over the years?
Huib
You are referring to materials I wrote within the Open Certification Project. I spent a few years on that. That project failed. It failed in a variety of ways. The underlying software-testing philosophy wasn’t wrong in principle, but some of our ideas about how to build sets of exam questions were completely impractical. I don’t see a path to a good “open certification” of the kind we imagined.
My practice is to acknowledge my failures, try to learn from them, and then try something else that is likely to be constructive and useful. In this case, we transferred the useful things that we learned from OCP to the BBST project. BBST is not inconsistent with the assessment values that I expressed in those documents. And I think BBST has been a much better use of my time and has offered more value to the community than the Open Certification Project could have provided.
Regarding ISTQB, I do not like high-stakes multiple-choice exams. It is possible to create good ones (I have seen some, such as the Multistate Bar Exam) but I do not think we can make this work well in software testing. I am not one of ISTQB’s supporters.
But I can believe that reasonable, honest people can hold a different opinion from me. That is not inconsistent with what I say in the posts you are citing. Even when I was actively pursuing the Open Certification Project, I also spoke strongly in support of accepting ASTQB as a sponsor at CAST conferences and when I spoke at CAST, I made a point of welcoming the sponsors, praising ASTQB for showing up at a meeting that was outside of their mainstream. I believe that a willingness and ability to learn from people who disagree with you is fundamentally consistent with the underlying philosophy of context-driven testing. And while I have not always practiced this perfectly (far from it), I believe that this has been an important part of my personal philosophy since I was 16.
As to asking ISTQB to back up its claims, yes of course. When ISTQB makes specific claims, it should have to back them up. But I am not aware of any evidence that this petition is directed toward any specific claims.
I see no inconsistency between disagreeing with ISTQB’s instructional methods and with its assessment methods while considering it inappropriate to demand their internal quality control records without strong reasons to tie those records to specific falsifiable claims that ISTQB is actually making.
— Cem Kaner
The ISTQB does not instruct anyone. The ISTQB is a not-for-profit organization based out of Belgium ostensibly tasks with the mission of developing a standardized body of knowledge against which it then provides assessment and certification services. The ISTQB does not, by itself, offer any training or coursework in order to help impart this BOK.
Pretty much every single member of the governing board, however, has a private for-profit company tasked with helping to prepare people for taking these examinations in order to receive their certification.
The petition is simply asking the not-for-profit company to defend its claims regarding their certifications and their testing/examination procedures. I have issues with their BOK but that is a different conversation. This is simply asking them to be internally consistent.
“My impression is that this petition is not a tool for holding ISTQB accountable for fraud.”
That’s why I didn’t sign it. And if fraud is being alleged, a petition is not the right avenue.
Personally I find ISTQB’s certification process laughable, but hey we’re all free thinkers and can decide where to spend our money.
“I believe that a willingness and ability to learn from people who disagree with you is fundamentally consistent with the underlying philosophy of context-driven testing.”
Yes! There is so much to that statement that seems to be getting lost these days.
Thanks for your post and the follow-up remarks.
Hi Cem,
Thanks for this post. I am not sure if ISTQB made any claims from its side but as a junior (less experienced in terms of years of IT) tester I would like to share what I have gone through and what I have been seeing around here in India.
In my previous organisation I was given this development goal to clear ISQTB certification to which I objected and asked to allow me to go for BBST. That was rejected. I do not know who is responsible for this but I have seen many people around me, let them be junior testers or senior managers who think that ISTQB is really some board which is equivalent to what other boards/bodies they have under UNESCO or NASA for that matter.
I am not sure who is responsible for this but by reading that 6 chapter book and answering those 40 exams (answers to which I had found controversial when I wrote foundation level) people start thinking that now they are expert testers and their ability to test can not be questioned. So is the case with Training institutes who are misguiding newbies to take ISTQB certification telling that they will get jobs only after that.
I am not sure who is responsible for this but my wife was rejected an interview call because her resume does not have that (prestigious?) logo of ISTQB certified. I believe that she understands testing well and she is still trying to find answer if that should really be a reason.
Let it be by training institutes, trainers , interviewing members, hiring managers and job consultancies….I am not sure who is responsible for this when they say ISTQB is ++ and No job/promotions if No ISTQB.
ISTQB might not have made any claims as such but I am not sure who is responsible who are making such rules, standards and assumptions on ISTQB’s behalf.
Reasons can be many but only thing I know is , this is certainly stopping testers from becoming ‘thinking testers’ and hence I found it worth to sign the petition. I am not sure if junior tester like me could question this or things mentioned above directly to ISTQB but this petition allowed me to.
I guess the “contex-driven testing” site should be renamed the “Cem Kaner House of Unnopposed Rants”…nice. It should also be noted that I have NEVER met or spoken to Cem Kaner. He has never reached out to me to discuss this topic or my views before launching his personal attacks on me – a courtesy I afforded the ISTQB and Rex Black.
I don’t know what has happened to you Cem (and I’m sure this will not get posted to what seems to be your personal blog now), but I think a comment posted the other day sums it up: Kaner’s rebuttal was a most-unfortunate example of principle betrayed. It will neither help him nor save the ISTQB.
Pingback: Perspectives on Testing » The Seapine View
Keith,
“I find it the height of irony when someone who has literally alienated themselves from the entire CDT community through paranoia and personal issues”
and
“you really should rename this URL or even shut it down, as it has become an embarrassment as a resource for the CDT community”
Do you honestly presume to speak for the whole CDT community? Please don’t – perhaps in your extensive circle of contacts there is disillusionment/disagreement on where this blog has gone or Cem’s position on certain matters…but there are those of us who don’t see any conflict or principles being violated.
Cem stated in a comment above:
“I see no inconsistency between disagreeing with ISTQB’s instructional methods and with its assessment methods while considering it inappropriate to demand their internal quality control records without strong reasons to tie those records to specific falsifiable claims that ISTQB is actually making.”
strikes me as fully reasonable, regardless of what one’s position is on ISTQB and certification in general.
What’s the real axe to grind here?
Pingback: On Testing Certifications, Knowledge, and Competence » The Seapine View
Pingback: On Certifications, Knowledge, and Competence | Cutting Edge Computing