Today is a typical day in the science news, because we see what we so often see: a press report claiming that neuroscientists have discovered something they have not actually discovered. The report is a press release from the University of Bonn with this headline: "'Math neurons' identified in the brain." Below this we have a subtitle reading, "When performing calculations, some neurons are active when adding, others when subtracting." While we know that humans can perform math calculations, there is no evidence that either brains or neurons perform math calculations. Guilty of serious methodological flaws, the scientific study in question has not found any good evidence that some neurons are more active than other neurons during math calculations.
The study (entitled "Neuronal codes for arithmetic rule processing in the human brain") can be read in full here. Nine epilepsy patients had electrodes attached to areas of their brains for medical reasons to determine the source of seizures they were having. Using such subjects, scientists attempted to find signs of greater activity in certain areas of the brain while the subjects performed a math task. Such a sample size of nine subjects was too small for a robust result. Fifteen subjects per study group is the minimum for a moderately persuasive result. When you use fewer than 15 subjects in a study group, there is a too high a chance of a false alarm.
The scientists recorded the electrical activity of about 600 neurons in each subject. They claim that a small percentage (about 5%) fired at a greater rate during addition or subtraction. But we would expect to get such a result by chance. Similarly, if I track for twenty minutes the minute-to-minute ups and downs of 600 stocks being traded on the New York Stock Exchange, and look for stocks that rise in price while I am thinking about cute puppies, I will probably be able to find that about 5% of the stocks seemed to have higher prices when I am thinking about cute puppies. This does nothing whatsoever to show that my thoughts about cute puppies have any influence on stock prices. I would in such a case have merely found a chance correlation that I would expect to get when comparing two unrelated things that fluctuate (stock prices and the objects of my attention). In all likelihood this is all that has turned up in the new "Neuronal codes for arithmetic rule processing in the human brain" study. The authors have merely found the type of chance correlation in electrical activity that we would expect to see in some small percentage of neurons (maybe 5% or so) when comparing the ups and downs of that electrical activity to something else that does not affect such electrical activity. No robust evidence has been provided of any causal effect.
Figure 2 of the paper is a line graph showing the ups and downs of the firing rate of four neurons, with two of the neurons showing increased activity during math calculation. The caption of the graph says it is showing activity for "four example neurons." When we remember that the electrical activity of about 600 neurons was tracked, we should not regard Figure 2 as being evidence for any causal effect. The authors probably cherry-picked some neurons out of their set of about 600, looking for a few with an electrical activity that seemed to rise during math calculation.
Similarly, if I did my experiment tracking the minute-to-minute price fluctuations of 600 stocks, while I was thinking about cute puppies, I could cherry-pick one stock with the strongest chance correlation, and produce a graph like the one below, similar to the graphs in Figure 2 of the paper.
How is it that we can judge whether a study like this has given robust evidence of anything (as opposed to showing only variations we would expect to get by chance)? We can look for 4 different things:
(1) Pre-registration. When pre-registration is used, scientists publicly pledge beforehand how data will be gathered and analyzed, reducing the chance the authors will be doing a kind of a "fishing expedition" in which they feel free to keep "slicing and dicing" the data dozens of ways until it seems to show the desired result: an approach that may be described as "keep torturing the data until it confesses."
(2) A blinding protocol. A blinding protocol is used to reduce the chance of experimental bias, an effect in which experimenters tend to see or find whatever result they hope to see.
(3) Control groups. When control groups are used, there are a group of subjects who do not receive the stimulus being applied to the main experimental group. The reaction of the group receiving the stimulus can be compared to a group that did not receive the stimulus.(4) Adequate sample sizes. An experiment should include a sample size calculation to determine the minimum number of subjects needed to provide robust evidence of a real effect. If such a calculation is not done, we should expect 15 subjects per study group as a minimum.
The new "Neuronal codes for arithmetic rule processing in the human brain" study fails on all these quality measures. The study was not a pre-registered study. The study failed to use any blinding protocol, and the paper does not use the word "blind" or "blinding." The sample size used (nine) is smaller than the minimum of 15 needed for a robust experimental result, and no mention is made of a sample size calculation. Although the paper uses the word "control" multiple times, the study did not use control groups. The use of a control group would have clarified that the main result reported is meaningless. In the control group we would have seen about 5% of neurons with increased activity when the subjects were not asked to do any math work. This would have helped make clear that the reported variations are merely chance variations, not actually evidence of "math neurons."
Being guilty of several methodological defects, the new "Neuronal codes for arithmetic rule processing in the human brain" study fails to provide any good evidence that there are "math neurons" in the brain, and fails to provide any good evidence of any such thing as "neuronal codes for arithmetic rule processing." Similar problems will be found in studies claiming to provide evidence for "time cells" and "place cells" in the brain, as discussed here and here.
The same type of methodological defects are found in another memory study released this week. Its press release groundlessly claims, "In a scientific first, researchers at the University of California, Irvine have discovered fundamental mechanisms by which the hippocampus region of the brain organizes memories into sequences and how this can be used to plan future behavior." A look at the Nature paper shows that a way-too-small sample size was used (only five rats), that no blinding protocol was used, that no control group was used, and that the scientists used some incredibly complicated "keep torturing the data until it confesses" approach that they presumably made up as they went along (since the paper makes no mention of pre-registering an exact hypothesis and pre-registering data collection and analytic methods).
A brain scan study looked for neural correlates of math calculation in adults and children, using a much better sample size of 20 adults and 80 children. As shown in Figure 2, the study found brain activity variations of only about 1 part in 200 or smaller, which is about what we would expect to have got purely by chance, even if the brain is not involved in calculating. The bar chart below puts such results in perspective.
No comments:
Post a Comment