What can be done?¶
I’ve discussed many statistical problems throughout this guide. They appear in many fields of science: medicine, physics, climate science, biology, chemistry, neuroscience, and many others. Any researcher using statistical methods to analyze data is likely to make a mistake, and as we’ve seen, most of them do. What can we do about it?
Statistical education¶
Most American science students have a minimal statistical education – perhaps one or two required courses, or even none at all for many students. And even when students have taken statistical courses, professors report that they can’t apply statistical concepts to scientific questions, having never fully understood – or simply forgotten – the appropriate techniques. This needs to change. Almost every scientific discipline depends on statistical analysis of experimental data, and statistical errors waste grant funding and researcher time.
Some universities have experimented with statistics courses integrated with science classes, with students immediately applying their statistical knowledge to problems in their field. Preliminary results suggests these methods work: students learn and retain more statistics, and they spend less time whining about being forced to take a statistics course.41 More universities should adopt these techniques, using conceptual tests to see what methods work best.
We also need more freely available educational material. I was introduced to statistics when I needed to analyze data in a laboratory and didn’t know how; until strong statistics education is more widespread, many students will find themselves in the same position, and they need resources. Projects like OpenIntro Stats are promising, and I hope to see more in the near future.
Scientific publishing¶
Scientific journals are slowly making progress towards solving many of the problems I have discussed. Reporting guidelines, such as CONSORT for randomized trials, make it clear what information is required for a published paper to be reproducible; unfortunately, as we’ve seen, these guidelines are infrequently enforced. We must continue to pressure journals to hold authors to more rigorous standards.
Premier journals need to lead the charge. Nature has begun to do so, announcing a new checklist which authors are required to complete before articles may be published. The checklist requires reporting of sample sizes, statistical power calculations, clinical trial registration numbers, a completed CONSORT checklist, adjustment for multiple comparisons, and sharing of data and source code. The guidelines cover most issues covered in Statistics Done Wrong, except for stopping rules and discussion of any reasons for departing from the trial’s registered protocol. Nature will also make statisticians available to consult for papers as needed.
If these guidelines are enforced, the result will be much more reliable and reproducible scientific research. More journals should do the same.
Your job¶
Your task can be expressed in four simple steps:
- Read a statistics textbook or take a good statistics course. Practice.
- Plan your data analyses carefully and deliberately, avoiding the misconceptions and errors you have learned.
- When you find common errors in the scientific literature – such as a simple misinterpretation of p values – hit the perpetrator over the head with your statistics textbook. It’s therapeutic.
- Press for change in scientific education and publishing. It’s our research. Let’s not screw it up.
