Here’s to a relatively recent TBT! In 2010, Craig Bennett and colleagues submitted a poster with the following title:
"Neural Correlates of Interspecies Perspective Taking in the Post-Mortem Atlantic Salmon: An Argument For Proper Multiple Comparisons Correction"
Yes, you are reading it right, it is about the neural correlates of a dead fish!
When laboratories buy a new fMRI scanner they need to test that it is working properly. Usually, they use a balloon filed with oil for this, but Bennett and his colleagues wanted to mimic the human head a bit more accurately. After trying a pumpkin and a dead Cornish Game Hen, they found our best equal to be an Atlantic Salmon, fresh from the supermarket. Just like our brains, a salmon’s flesh is very fatty, and it has skin and bones to nicely resemble the rest of our heads!
They put the fish in the scanner and presented him with the experiment they were subjecting their human participants to: watching pictures of human faces and determining their emotional expression.
They scanned the fish during the experiment and analyzed his BOLD signal (fMRI scanners measure brain activity by looking at changes in oxygenized blood levels, active brain regions attract a greater blood flow to them than inactive brain regions). They divided the scan of the salmon in little cubes, called voxels and measured BOLD changes in each voxel.
Excitingly, this analysis brought out some active voxels! And even more promising, the voxels were located in the fish’s brain….
Now, this study was not as interesting as you might have hoped: the dead salmon was not responding to emotional faces, even though Bennett’s analysis clearly showed activity in his brain. However, it was a highly exhilarating and, at the same time concerning, study for the world of neuroscience:
Until Bennett showed that even a dead fish can give you significant results, many labs were using exactly the analysis Bennett applied to his data set. They took all their 130.000 voxels and tested each of them. With such a high amount of tests, it is almost certain that one of them will (falsely) come out positive!
This phenomenon is called the multiple comparisons problem. A correct analysis would have corrected for the false positive rate and would not have shown any active voxels in the ceased fish’s head. However, until 2010, 25-40% of the studies did not make this correction, meaning that many studies might have found results that were as true as a dead fish! Luckily, this number has dropped severely since Bennet's study and many other papers that stressed the importance of correcting for multiple comparisons.
I thought of this study because I ran into a slightly similar problem last week:
I was recording brain activity from a mouse that I had taught to expect reward (a drop of tasty water) after smelling certain odors. As it turned out, I had not plugged in my recording system correctly, but I did not know that at that time.
When I looked at the activity on my screen I saw a signal that beautifully increased for odors that predicted reward and not for odors that did not predict reward. Even more, it peaked at the time of reward delivery, which was very exciting to me since I study the development of neural activity over time.
For 5 minutes I was in heaven, thinking I had finally found the signal was looking for. Then I found out that the activity was actually perfectly aligned to the mouse’s licking behavior. In fact, a little too perfectly. It turned out that what I was seeing was simply an artifact: the mouse’s licking movements made my recording system move, which looked exactly like the neural activity I was looking for.
Just like the salmon paper, this moment proved to me that you can almost always find what you are looking for, be it due to too many statistical tests or an experimental artifact. Staying objective and doing all the necessary corrections and controls is of crucial importance for good scientific work. But it is also hard. When you have found what you are looking for you don’t want to throw it straight back into the trash!
In the past years, a couple of important studies have been retracted because the results turned out to be simply untrue. With increasing competition in the scientific world, the pressure to publish and thus the pressure to find interesting results are growing to problematic levels. It is hard to stay objective if your career depends completely on this particular result. (See also Nick’s blog post about this)
Luckily, many people in science see this danger and some initiatives have been started to clean up the literature and encourage critical and objective work (see for example, retraction watch and the center for open science. Of course, the competitive environment is the real problem, but that will need some more work, will and time to be solved.
And so that was the story of the dead fish. In 2012, Craig Bennett was awarded the IgNobel Prize in Neuroscience. Thanks to his study, false positives will now forever remind us (or at least me) of Atlantic Salmon!
Craig M. Bennett, Abigail A. Baird, Michael B. Miller, George L. Wolford. (2010).Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon: An argument for multiple comparisons correction. Journal of Serendipitous and Unexpected Results.