On the Petition to ISTQB


I don’t usually let myself get dragged into the TwitterStorms orchestrated by the Rabid Software Testing crowd. But I decided to make an exception for this one because it has sucked some good people into endorsing what I think is a very bad petition.

In essence, the petition asks ISTQB to open its quality-control records. ISTQB is involved in training and certifying software testers, so I will write this to you (the reader) as if you are a software tester. The basic question is whether ISTQB has seen any problems in their exams and what they have done about it. (Underneath that are some more specifically-pointed details, which I’ll come to later.)

Let’s Start On This From The Basic Principle

In many countries (such as the United States), companies have a right to study the quality of their goods or services in private. That’s what I understand that ISTQB has been doing.

By the way, that’s what companies do when they test their software. Those companies (ISTQB, and software companies) have a right to hire employees and consultants and to require those people to keep what they’ve learned private. When you test your company’s or client’s software, it is normally a condition of your job or your consultancy that you keep your results private.

This petition asks ISTQB to do what (for the most part) the companies you work for would not do, and would not dream of letting you do with their quality-control data.

When I studied law, I learned that the public policy of the United States (and many other countries) favors investigative privilege. That is, we encourage companies to conduct aggressive, detailed internal investigations, to find their own problems and to improve their products, their services and their advertising based on what they learn from the investigation. We want them to investigate because we want the improvement.

These types of studies expose the companies to risk–people who get their hands on the investigative data, and don’t like those company that investigates itself, can use any report of any problem, whether they understand the problem or not, whether the problem is actually major or not, whether the company has dealt with the problem in a reasonable way or not–to attack the company. Therefore, to encourage companies to do the studies, society grants them privacy–we treat the studies as trade secret–because the companies won’t do serious studies without privacy.

When I practiced commercial law, my most significant project involved drafting legislation about the law of software quality. My most significant contribution was a very narrow disclosure rule that would preserve software vendors’ incentives to do internal investigations (i.e. test their own software harshly) while holding them accountable for significant defects that they knew about but did not make public. It took 6 years to craft the language for this. It took 6 years because the idea of almost-forced disclosure of (some) test results triggered allergic reactions among lawyers who normally defended companies, among lawyers who normally attacked companies, and among judges. Finding a way to balance the interests here was enormously difficult because the support for allowing companies to investigate their products and services in private runs so very deep. And, in my view, it should.

And if you want to be able to do significant work as testers, you should support that privilege too.

And that means you should show some respect for that privilege, even when the company asserting it is a company that you (or some people who impress you by screaming at others in public) don’t like.

Now, we have a petition from someone who frequently attacks ISTQB, asking them to open their private quality-control records associated with their certification exams. Will I sign it?

No, Keith. I will not sign that petition.

A Little More Background

Back in 2010, I saw some materials that allegedly summarized unauthorized disclosures from an ISTQB meeting. I suspect that this (2010) material underlies the specificity of the questions in this (2013) petition to ISTQB.

The materials were not authenticated and there might not be any truth in them at all. The gist of the materials was that ISTQB had commissioned a psychometric study of one or more of their tests, that a “test reliability coefficient” was a bit low, and that ISTQB was planning to use this information to improve the quality of their exams.

I believe that this is the type of research that we, as a society (and as a profession of testers), want to see done. Look for problems. When you find problems, raise them with other decision-makers in the organization and look for ways to improve them.

To keep this going, society respects these studies as trade secret–grants them privacy. There is no expectation, zero, none, that they will tell us about the studies or what they did to address them.

(There are some exceptions to that rule. I will note them later, to suggest that they are probably not relevant here.)

Certain people encouraged me to get involved in attacking ISTQB with these allegations. I refused.

One thing I said to those people, back in 2010, was:

“I’m not a huge fan of attacking competitors. I fall into it, but I think it is a distraction. I think it adds to the personality circus of the field and detracts from the content of the field.”

Another of the things I said was:

“As to the ethics, I am slightly outraged by the fact that ISTQB apparently will not make this public. But I am also slightly outraged that this was leaked, if it was. You’re being offered the opportunity to play the role of one hostile competitor viciously attacking another competitor. Your attack will not look ethical.”

A little later, I looked more carefully at this “test reliability coefficient”. I am not an expert in this area of statistics, but I know a little bit.

  • My impression is that the theory underlying the meaning and application of the statistic is not fully developed.
  • My impression is that the cutoff that separates “good” values from not-good-enough values is arbitrary.
  • My impression is that something like this is probably useful for internal studies, because it can raise flags. A not-excellent number can motivate people to improve, even if that number is not necessarily very bad, and even if the other aspects of the meaning of the number are a little uncertain.

I’ve dealt with lots of metrics like this in my career. They are good enough for private, informal, self-evaluation or evaluation by a friendly coach. But the idea of attacking a company based on not-terrible-but-not-great values of these statistics, well, I don’t think that’s a valid use of this statistic. Not without a whole lot of other converging evidence.

You may have seen me quoted about these types of metrics once or twice. The quote goes:

“Metrics that are not valid are dangerous.”

So, once I studied the statistic a little, I was no longer even a little bit outraged that ISTQB wasn’t making the data public. It was not only their privilege to keep it private. Keeping it private was also, in my view, reasonable.

I think that using metrics like this to publicly attack people (or companies) is unethical. Even if they are people or companies that I disagree with, or don’t like.

So, Keith, No, Keith. I will not sign that petition.

The Big Public Attack on ISTQB over this didn’t happen in 2010 and I expected never to hear of these allegations again.

Should We Ever Force Companies To Disclose Private Data?

Sometimes, society forces companies to disclose their private data.

For example, if a company commits crimes (you know, like rigging LIBOR, or engaging in tax evasion, or laundering money), its victims (and prosecutors representing society as a whole) can demand to see relevant records and the courts will enforce the demand.

Suppose that ISTQB made specific comments about the reliability of their tests, and they were in possession of data that indicated that those specific comments were false. That would be fraud. In that case, it should be possible to obtain ISTQB’s internal data.

I’m not a big fan of ISTQB’s syllabus or exams, or of their marketing. My attitudes are not as extreme as some other people’s. Some twits seem to believe that ISTQB’s materials are worthless, that everyone finds them worthless, and that no reasonable person thinks the exams or the certifications are worth anything.

  • I have been told by some reasonable people that they believed they learned a great deal from ISTQB courses.
  • I have been told by some reasonably bright people that they found an ISTQB exam challenging.
  • I have been told by some teachers that they see value in the material.
  • I have been told by some hiring managers who have testing experience that they believe that this should be a positive hiring factor.

That’s not exactly my evaluation, but I have no reason to doubt the integrity of these other people, or doubt their experience.

But because I’m skeptical of ISTQB, every now and again, I look at their marketing materials. I haven’t looked at everything. I haven’t even looked at most of the materials. But in what I have seen, even though I don’t agree with the views being expressed or the conclusions they would have me draw:

  • I have not seen evidence of fraud
  • I have not seen claims that could be refuted by mediocre results in test reliability scores.

So. Keith. No, Keith. Absolutely not, Keith. I will not sign that petition.

I encourage you to withdraw it.


11 thoughts on “On the Petition to ISTQB

  1. While I remain unimpressed by the bluster of @RBCS, I am inclined to agree with you, sir: no good purpose will be served by this petition effort.

  2. Hi Cem,

    Interesting post. I have to admit I do not know a lot about law so I will immediately believe you when you say it is legal not to disclose private data. But is it ethical to do so? If an organisation makes claims openly, it sounds only reasonable to me if people ask for clarification and evidence to back those claims up.

    When I read these documents (http://kaner.com/pdfs/ocp1.pdf and http://kaner.com/pdfs/ocp2.pdf) on your website it seems to me that you have changed your mind. Am I right?

    In these documents you say: “Software testers are professional skeptics. To require them to adopt a compliance mentality, in which they set aside issues of ambiguity, oversimplification, unstated assumptions or controversial conclusions in order to provide the answer expected by an examiner is to demand conduct so far removed from what testers should do as to be invalid on its face.” and “The emphasis on openness makes possible a more revealing and less prescriptive use of certification exams.”

    These statements seem not consistent with what you say in this post. What made you change your mind over the years?

    • Huib

      You are referring to materials I wrote within the Open Certification Project. I spent a few years on that. That project failed. It failed in a variety of ways. The underlying software-testing philosophy wasn’t wrong in principle, but some of our ideas about how to build sets of exam questions were completely impractical. I don’t see a path to a good “open certification” of the kind we imagined.

      My practice is to acknowledge my failures, try to learn from them, and then try something else that is likely to be constructive and useful. In this case, we transferred the useful things that we learned from OCP to the BBST project. BBST is not inconsistent with the assessment values that I expressed in those documents. And I think BBST has been a much better use of my time and has offered more value to the community than the Open Certification Project could have provided.

      Regarding ISTQB, I do not like high-stakes multiple-choice exams. It is possible to create good ones (I have seen some, such as the Multistate Bar Exam) but I do not think we can make this work well in software testing. I am not one of ISTQB’s supporters.

      But I can believe that reasonable, honest people can hold a different opinion from me. That is not inconsistent with what I say in the posts you are citing. Even when I was actively pursuing the Open Certification Project, I also spoke strongly in support of accepting ASTQB as a sponsor at CAST conferences and when I spoke at CAST, I made a point of welcoming the sponsors, praising ASTQB for showing up at a meeting that was outside of their mainstream. I believe that a willingness and ability to learn from people who disagree with you is fundamentally consistent with the underlying philosophy of context-driven testing. And while I have not always practiced this perfectly (far from it), I believe that this has been an important part of my personal philosophy since I was 16.

      As to asking ISTQB to back up its claims, yes of course. When ISTQB makes specific claims, it should have to back them up. But I am not aware of any evidence that this petition is directed toward any specific claims.

      I see no inconsistency between disagreeing with ISTQB’s instructional methods and with its assessment methods while considering it inappropriate to demand their internal quality control records without strong reasons to tie those records to specific falsifiable claims that ISTQB is actually making.

      — Cem Kaner

  3. The ISTQB does not instruct anyone. The ISTQB is a not-for-profit organization based out of Belgium ostensibly tasks with the mission of developing a standardized body of knowledge against which it then provides assessment and certification services. The ISTQB does not, by itself, offer any training or coursework in order to help impart this BOK.

    Pretty much every single member of the governing board, however, has a private for-profit company tasked with helping to prepare people for taking these examinations in order to receive their certification.

    The petition is simply asking the not-for-profit company to defend its claims regarding their certifications and their testing/examination procedures. I have issues with their BOK but that is a different conversation. This is simply asking them to be internally consistent.

    Thanks for your note.

    Yes, ISTQB is a non-profit corporation that provides assessment services.

    If ISTQB is making specific, falsifiable claims regarding their certifications and testing examination procedures, and if the proponent(s) of the petition believe they have (or could get) data that show that those claims are fraudulent, I think they should go through the legal process. They can get plenty of data through the legal process, without having to say please or run a public petition. I am a huge fan of prosecuting consumer fraud.

    My impression is that this petition is not a tool for holding ISTQB accountable for fraud. My impression is that it is just another public relations stunt, another vehicle for some people to attack ISTQB.

    In my post, I tried to explain why I believe that some types of attacks do more harm than good. Even if my specific impressions about this particular attack are mistaken, I still think that this is an example of the kinds of attacks that I think of as toxic. I also tried to explain why testers should be particularly sensitive to the societal tradeoffs involved in these types of attacks.

    — Cem Kaner

  4. “My impression is that this petition is not a tool for holding ISTQB accountable for fraud.”

    That’s why I didn’t sign it. And if fraud is being alleged, a petition is not the right avenue.

    Personally I find ISTQB’s certification process laughable, but hey we’re all free thinkers and can decide where to spend our money.

    “I believe that a willingness and ability to learn from people who disagree with you is fundamentally consistent with the underlying philosophy of context-driven testing.”

    Yes! There is so much to that statement that seems to be getting lost these days.

    Thanks for your post and the follow-up remarks.

  5. Hi Cem,

    Thanks for this post. I am not sure if ISTQB made any claims from its side but as a junior (less experienced in terms of years of IT) tester I would like to share what I have gone through and what I have been seeing around here in India.

    In my previous organisation I was given this development goal to clear ISQTB certification to which I objected and asked to allow me to go for BBST. That was rejected. I do not know who is responsible for this but I have seen many people around me, let them be junior testers or senior managers who think that ISTQB is really some board which is equivalent to what other boards/bodies they have under UNESCO or NASA for that matter.

    I am not sure who is responsible for this but by reading that 6 chapter book and answering those 40 exams (answers to which I had found controversial when I wrote foundation level) people start thinking that now they are expert testers and their ability to test can not be questioned. So is the case with Training institutes who are misguiding newbies to take ISTQB certification telling that they will get jobs only after that.

    I am not sure who is responsible for this but my wife was rejected an interview call because her resume does not have that (prestigious?) logo of ISTQB certified. I believe that she understands testing well and she is still trying to find answer if that should really be a reason.

    Let it be by training institutes, trainers , interviewing members, hiring managers and job consultancies….I am not sure who is responsible for this when they say ISTQB is ++ and No job/promotions if No ISTQB.

    ISTQB might not have made any claims as such but I am not sure who is responsible who are making such rules, standards and assumptions on ISTQB’s behalf.

    Reasons can be many but only thing I know is , this is certainly stopping testers from becoming ‘thinking testers’ and hence I found it worth to sign the petition. I am not sure if junior tester like me could question this or things mentioned above directly to ISTQB but this petition allowed me to.

    I think there are many good reasons to be annoyed with ISTQB and to question the value of their certifications. I can imagine many petitions that address ISTQB in a critical way that I would be willing to sign (or at least to not oppose).

    However, I think there are limits on the ways that we should attack each other. This is what I was getting at when I referred to the “rabid software testers.” I think a group in our community is engaging in grossly uncivil behavior. I have no insight into their internal mental processes, but from their actions I form the impression that they are more motivated to attack people they perceive as enemies and to intimidate people who might disagree with them than they are to consider the long-term consequences on our field of their actions.

    I think some types of actions can cause long-term harm even if they offer the potential of short-term advantage.

    When I first heard the rumour that ISTQB was hiring psychological researchers to investigate the quality of their testing, I was pleased that they were doing this. When I heard the rumour that they were discussing the results at a Board meeting, because they allegedly were unhappy with the results and were going to fund process improvements to improve the results, my respect for ISTQB increased. The rumour specifically identified Rex Black as the initiator of the research and the initiator of the discussions in the Board. Because I believed the rumour, my respect for Rex increased substantially.

    Please, think like a tester. People hire us to find bad things about their products. We tell them. And they (hopefully) fix those bad things. Many of us get deeply annoyed by companies that discourage us from taking too close a look or from reporting problems that we see as significant. Our clients would not hire us, or they would restrict our testing significantly, if they believed that everything we found could be discussed openly on the Net. Our freedom to do skilled work as individual testers and our business models as independent test labs depend on our clients’ confidence that our findings will stay private, that they can improve their products without fear of public embarrassment from our work.

    This is the investigative privilege. The courts can pierce it, ordering a company to open its records. However, such an action requires good cause, such as a plausible lawsuit for fraud or a government (regulatory or criminal) investigation.

    My argument is that as a society, we should be very cautious about attempting to pierce this privilege because most product and service problems are found and fixed via internal investigations. My impression is that the belief in the value of the investigative privilege is widespread among people who have thought deeply about the policies that underlie laws governing fraud, defects, and breaches of contract. I have been deeply impressed by the extent to which senior people, who otherwise have sharply different views about those policies, agree on this. They (and I) believe that attacking companies for their quality-control findings will encourage them to do much less quality-control research (i.e. less testing) and therefore they will find less and they will fix less.

    The investigative privilege applies as much to nonprofit companies (like ISTQB) as for-profit ones, and to individual businesses (e.g. one-person consultancies) as to large corporations. The privilege applies as much to weaknesses in services (e.g. exams) as to defects in products. It is the same underlying policy in each case — we want people to investigate their products and services aggressively so that they can find their problems and fix them.

    I think it is generally a bad idea to attack this policy. I think it is a Really Bad Idea for testers to attack it. Our clients depend on this privilege and they depend on us to collaborate with them to protect the secrets that this privilege entitles them to protect. When we attack that privilege (which is what the petition and the associated attacks on Rex and ISTQB are doing), we give our clients good reason to wonder whether they can afford to trust our commitment to that privilege in the future, as it will apply to others (like them).

    A different issue that I have with this petition is that (I believe) it is focused on a rumoured value of a specific metric that I don’t think is a very good metric. I think it is outrageous — not surprising, but still outrageous — for people who rail against the use of metrics, who posture that it is deeply unethical to even calculate or report the values of questionable metrics — for THESE people to attack a company based on an allegedly-borderline value of a questionable metric.

    I think that bad metrics are bad even when they can be used to attack people I don’t like. I think that societal privileges that shield people must apply even when they shield people that I don’t like.

    For the first few years of ISEB and ISTQB, I think it was reasonable (in retrospect, it was not a wise use of time, but I think it was reasonable) to attack ISEB’s / ISTQB’s model. By 2007, it was clear that this wasn’t working. The way to beat ISTQB is not by screaming at them, not by blaming them for the weaknesses in our profession and in the perception of our profession, but by offering other credentials that are better and other training that supports those credentials in ways that foster more skill and more-critical thinking. In 2007, in the same grant proposal that funded the development of BBST, Rebecca Fiedler and I wrote “Certification is an application of our core project, [but] not the driver of it.”

    Seven years later, we don’t have good sets of credentials that can be used as alternatives to ISTQB. But I think some are gradually coming. Ranting against ISTQB is easy. Doing the work needed to create better alternatives is harder, but I think it will be more constructive for the field (testers will learn more about testing and less about anger and rudeness).

  6. I guess the “contex-driven testing” site should be renamed the “Cem Kaner House of Unnopposed Rants”…nice. It should also be noted that I have NEVER met or spoken to Cem Kaner. He has never reached out to me to discuss this topic or my views before launching his personal attacks on me – a courtesy I afforded the ISTQB and Rex Black.

    I don’t know what has happened to you Cem (and I’m sure this will not get posted to what seems to be your personal blog now), but I think a comment posted the other day sums it up: Kaner’s rebuttal was a most-unfortunate example of principle betrayed. It will neither help him nor save the ISTQB.

    Keith. It’s not about you. It’s about behavior.

    Context-driven testing was something that I helped create and something that I put a huge amount of work into. I think it offers a lot of potential value to the testing community, but not if it is used as a bludgeon rather than as a guide. Many people have commented about their fear of bullying or their bad reaction to it or their bad reaction to what they see as an increasing level of incivility in our community. I think this has gotten more intense over time.

    When I see a set of positions, claims or ideas that I think are destructive being marketed as “context-driven”, and when I see some indication that people in the community are being persuaded to think these are actually consistent with “context-driven testing” then I consider speaking out about those positions, claims or ideas. That includes pointing out their problems and also explaining how I think a context-driven approach would operate instead.

    Should I do this? People will reach their own conclusions. For me, the relevant teaching is “If not me, who? And if not now, when?” (Hillel)

    It is laughable to see my comments interpreted as an effort to help Rex or save ISTQB.

    My main current comment on certification is at http://kaner.com/?p=317. I expected this to be my only comment on certification for the next few months.

    The petition drive and associated stream of comments on Twitter were things I had planned to ignore as a negative-energy distraction. But then you started using Twitter to publicly press individuals to comment on this petition. You didn’t send me a private email to see if I could be recruited to publicly comment on your campaign. (If you had, I would have politely ignored it or politely-and-privately-to-you responded by declining to have anything to do with it.) Instead, you made a public call for my response on Twitter. OK, at your specific, published request, I responded to you. Sorry you didn’t like it. It is what you asked for.

    — Cem Kaner

  7. Pingback: Perspectives on Testing » The Seapine View

  8. Keith,

    “I find it the height of irony when someone who has literally alienated themselves from the entire CDT community through paranoia and personal issues”


    “you really should rename this URL or even shut it down, as it has become an embarrassment as a resource for the CDT community”

    Do you honestly presume to speak for the whole CDT community? Please don’t – perhaps in your extensive circle of contacts there is disillusionment/disagreement on where this blog has gone or Cem’s position on certain matters…but there are those of us who don’t see any conflict or principles being violated.

    Cem stated in a comment above:

    “I see no inconsistency between disagreeing with ISTQB’s instructional methods and with its assessment methods while considering it inappropriate to demand their internal quality control records without strong reasons to tie those records to specific falsifiable claims that ISTQB is actually making.”

    strikes me as fully reasonable, regardless of what one’s position is on ISTQB and certification in general.

    What’s the real axe to grind here?

  9. Pingback: On Testing Certifications, Knowledge, and Competence » The Seapine View

  10. Pingback: On Certifications, Knowledge, and Competence | Cutting Edge Computing

Comments are closed.