I read a paper today in the British Medical Journal presenting a compelling case for a randomized controlled trial to measure the effectiveness of parachute use as a medical intervention to prevent injury when jumping from an aeroplane (Smith & Pell, 2003).
Their reasoning was as follows:
- No such trial has been done before
- The perception of the success of the parachute is based solely on anecdote
- Natural history studies show that free fall without does not always result in an adverse outcome, therefore there is not a 100% chance of mortality when not using a parachute
- There is also not a 100% chance of survival when using a parachute
- In all other medical interventions not having a randomized controlled trial would be unacceptable
- Current industry trials are subject to selection and reporting bias, as well as conflicts of interest.
All in all, there seems to be plenty of reasons why a randomised controlled trial should be carried out, so as to determine more appropriately the efficacy of the parachute… Except for the fact it would be stupid. And that, leads me on to the post-mortem Atlantic salmon.
Bennet, Baird, Miller and Wolford (2009) compared fMRI results of a dead Atlantic Salmon to a group of normal participants in a social perspective taking task.
Image Credit: Bennet, Baird, Miller and Wolford, 2009
The blood-oxygen-level-dependent (BOLD) signal experienced a significant change during the presentation of stimuli compared to at rest (t(131) > 3.15, p(uncorrected) < 0.001). An 81mm3 cluster of active voxels was identified on the Salmons brain cavity, and another smaller cluster was identified in the Salmons dorsal spinal column.
After the same t-tests were done using Benjamimi-Hochberg and family-wise corrections, both with a type 1 error rating of 0.05, no significant results were found. With those error ratings adjusted to 0.25, there was still no brain activity in Mr Salmon.
So, we have two daft studies here, both of which tickled me. It shows the importance of the correct use of advanced (and less advanced) statistics. It’s important to have a firm grasp of the statistical techniques you are working with, or you can very easily end up with errors. This is especially so in the case of the Salmon where such a large number of fine readings are recorded that without an element of realism being considered during the analysis, obvious statistical errors pop up.
And of course, the same level of realism is required at the opposite end of the spectrum, when you ask if we really need experimental control at all. I am impressed with how statistics allow researchers in the social sciences to measure population variables with precision. I don’t even pretend for one minute that I have an in depth understanding of all the methods covered even at undergraduate level, but I have learnt the value of getting stats right. It is important to properly reflect on what statistical tools are genuinely necessary to answer the research question, and of the ones you choose to use, it is important to understand the intricate details.