The Cult of Statistical Significance

If you see statistical significance or significance testing in your Employee , Customer or Operations Research  then in all likelihood you are doing it wrong. Equally important, conclusions drawn and decisions made on the basis of significance tests may very well be doing you more harm than good.

This is the basic message of  The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice and Lives, by Stephen T. Ziliak and Deirdrie McCloskey which has just hit the stores. Well, given that this is a rather technical book on the impact of statistical significance testing, just hit the stores may be a little strong. Just hit Amazon might be more accurate.

Nevertheless, for the technically minded, The Cult of Statistical Significance is a goldmine of information detailing the misuse of statistical significance (and significance testing). It provides a rich history of the technical advances made in the development of statistical theory as well as some of the politics that influenced where we are today. Unfortunately, where we stand today is in a field characterized by widespread misunderstanding and  misuse of statistical significance as a measure of practical importance.  It is an error that, from the book’s subtitle, costs us jobs, justice and lives.

It is also an error we are all too familiar with. As our Voice of the Customer and Voice of the Employee clients are well aware, Converge has been warning against the misapplication of significance testing in this regard for twenty years. We never use statistical significance as a measure of practical importance  in any of our survey research projects.

Both our Voice of the Customer and Voice of the Employeeproducts emphasize highlighting things of practical importance — a measure of real world significance or what authors Ziliak and McCloskey refer to as ooomph.

Calculating statistical significance is easy — too easy. To give third rate research the look of quality, all you have to do is click the correct button on your statistical software package. Real knowledge never comes that easily.

Determining what is of real world significance, in contrast, is difficult. It is, nevertheless, the valuable part of research. Businesses are not typically interested in calculating probabilities in accordance with a probability model. They are more likely interested in determining what is of real importance to customers, employees or to the performance of processes. Identifying those problems that are having a real impact on the business and developing evidenced-based solutions to address these problems is likely more important to management (and certainly to the business) than playing games with the development null and alternative hypotheses. 

The Cult of Statistical Significance provides a detailed and well documented demolition of the use of statistical significance testing and a brilliant argument in support of practical importance as the critical test in survey, or any other type of research.

In reviewing the book, Thomas Schelling, Distinguished University Professor, School of Public Policy, University of Maryland and 2005 Nobel Prize Laureate in Economics had this to say:

McCloskey and Ziliak have been pushing this very elementary , very correct, very important argument through several articles over several years and for reasons I cannot fathom it is still resisted. If it takes a book to get it across , I hope this book will do it. It ought to.”

If you enjoyed this post, please consider leaving a comment or subscribing to the RSS feed to have future articles delivered to your feed reader.

Filed Under: Survey ResearchVoice of the CustomerVoice of the Employee

About the Author:

RSSComments (1)

Leave a Reply | Trackback URL

  1. […] has had a profoundly negative impact on organizations as we have documented previously (see The Cult of Statistical Significance). Organizations misinterpreting statistical significance simply end up addressing the wrong things […]

Leave a Reply

    Why products get better and service gets worse

    Posted on: 2012-07-25

    Service standards are really a means of avoiding accountability CALGARY, AB, Jul 17, 2012/Troy Media/ – Remember when getting a 100,000 miles of your car was considered a miracle of sorts? Those invited to bear witness performed the ritual of peering into the driver side window to confirm all those zeros on the odometer. The

    Read More…

- More From Our Blog-