"Thousands of Participants Are Needed for Accurate Results," But Most Brain Scan Studies Don't Even Use Dozens

For many years neuroscientists have been claiming important results about brains and minds, after doing brain imaging experiments using very small sample sizes.  For example, we may read headlines saying that some particular region of the brain is more active during some type of mental event,  and the total number of subjects who had their brains scanned will usually be smaller than 15. A new press release from the University of Minnesota Twin Cities announces results which indicate that such small-sample correlation-seeking brain imaging experiments are utterly unreliable.  The headline of the press release is "Brain studies show thousands of participants are needed for accurate results."

We read this:

"Scientists rely on brain-wide association studies to measure brain structure and function—using MRI brain scans—and link them to complex characteristics such as personality, behavior, cognition, neurological conditions and mental illness. New research published March 16, 2022 in Nature from the University of Minnesota and Washington University School of Medicine in St. Louis...shows that most published brain-wide association studies are performed with too few participants to yield reliable findings."

The abstract of the paper in the science journal Nature can be read here. The paper is entitled, "Reproducible brain-wide association studies require thousands of individuals." 

The press release tells us this:

"The study used publicly available data sets—involving a total of nearly 50,000 participants—to analyze a range of sample sizes and found:

• Brain-wide association studies need thousands of individuals to achieve higher reproducibility. Typical brain-wide association studies enroll just a few dozen people.
• So-called 'underpowered' studies are susceptible to uncovering strong but misleading associations by chance while missing real but weaker associations. 
• Routinely underpowered brain-wide association studies result in a surplus of strong yet irreproducible findings."

The claim that a typical brain scanning experimental study uses "a few dozen" people is probably an overestimate. Brain imaging studies touted in the press seem to typically involve fewer than 15 subjects. 

The press release tells us that the conclusions above are based on some very heavy number crunching using databases that store brain scans of a large number of people, including in many cases data on what they were doing or thinking while being scanned, what kind of mental characteristics or health history the people had, and what kind of genes the people had.  The largest such database was the UK Biobank, which according to page 5 of the document here includes "resting-state functional MRI measures changes in blood oxygenation associated with intrinsic brain activity (i.e., in the absence of an explicit task or sensory stimulus)," as well as "task-functional MRI" which "uses the same measurement technique as resting-state fMRI, while the subject performs a particular task or experiences a sensory stimulus." (The task was mainly something called the Hariri faces/shapes “emotion” task.)  Another large database used was a Human Connectome Project database including "task-evoked fMRI" brain scans of people while they were doing things involving working memory, gambling, language, social cognition, relational processing and emotional processing (as mentioned on page 36 of the document here).  Another large database used was an Adolescent Brain Cognitive Development (ABCD) database that included fMRI scans while subjects performed tasks such as a Monetary Incentive Delay task. a Stop Signal task and an "n-back" or "nBack" task (as described here). 

In the press release we read this:

"To identify problems with brain-wide association studies, the research team began by accessing the three largest neuroimaging data sets: the Adolescent Brain Cognitive Development Study (11,874 participants), the Human Connectome Project (1,200 participants) and the UK Biobank (35,375 participants). Then, they analyzed the data sets for correlations between brain features and a range of demographic, cognitive, mental health and behavioral measures, using subsets of various sizes. Using separate subsets, they attempted to replicate any identified correlations. In total, they ran billions of analyses, supported by the MIDB Informatics Group and the powerful computing resources of the Minnesota Supercomputing Institute. The researchers found that brain-behavior correlations identified using a sample size of 25—the median sample size in published papers—usually failed to replicate in a separate sample.  As the sample size grew into the thousands, correlations became more likely to be reproduced. Robust reproducibility is critical for today’s clinical research. Senior author Nico Dosenbach, MD, PhD, an associate professor of neurology at Washington University, says the findings reflect a systemic, structural problem with studies that are designed to find correlations between two complex things, such as the brain and behavior."

What this study very strongly indicates is that the vast majority of brain imaging studies trying to correlate brains and mental states or mental activity have misled us by producing false alarms. The study indicates that such brain imaging studies have not merely been guilty of some slight shortfall, but have been guilty of a hundred-fold shortfall (the difference between about 20 and "thousands" being a difference of a hundred times).  It's as bad as if someone told you he produced a score of 1000 on his SAT test, but really only produced a score of 10. 

The study described above was led by neuroscientist Scott Marek. An article on the study in the journal Nature says this:

“ 'There’s a lot of investigators who have committed their careers to doing the kind of science that this paper says is basically junk,' says Russell Poldrack, a cognitive neuroscientist at Stanford University in California, who was one of the paper’s peer reviewers. 'It really forces a rethink.' ”

The New Scientist article on the Marek study is behind a paywall, but at least I can show its headline:

For many years we have been scammed and the US federal government has been scammed by neuroscientists doing ridiculously low-powered brain imaging studies looking for correlations between brains and minds.  For many years our experimental neuroscientists doing small-sample brain imaging studies (looking for correlations between brain states and mental states) have been playing a game of "sham, scam, thank you Sam," the Sam being Uncle Sam who provided the dollars for such worthless studies producing only false alarms. This is a racket, but since it is a nice little source of dishonest income and easy work for professors, the racket will probably long continue. 

The US government seems to be incredibly poor at recognizing bad performance by biology authorities.  In the New York Times there was recently an opinion article with the headline "How Millions of Lives Might Have Been Saved from COVID-19." Without naming any names of the bumbling officials guilty of the bungled US response to COVID-19, we get some startling comparisons between competent responses in small countries and incompetent responses in the US. For example, we are told that Taiwan has suffered only 853 COVID-19 deaths, and that "if the United States had suffered a similar death rate, we would have lost about 12,000 people, instead of nearly a million."  Because the US government seems to be extremely poor at recognizing bad performance by biology authorities. we will probably continue to see the "sham, scam, thank you Sam" researchers bilking the government by doing worthless federally-funded small-sample brain imaging studies producing only  false alarms. 

One of the quotes above tells us that correlations reported with a sample size of 25 "usually failed to replicate in a separate sample," but that "as the sample size grew into the thousands, correlations became more likely to be reproduced." Does this mean that strong correlations were found between brains and cognitive activity or cognitive states as long as you used samples of thousands? No. The Nature article on the Marek study tells us this:

"Researchers measure correlation strength using a metric called r, for which a value of 1 means a perfect correlation and 0 none at all. The strongest reliable correlations Marek and Dosenbach’s team found had an r of 0.16, and the median was 0.01."  

So even when data on thousands of subjects was used, no strong or medium correlations were found, and the median correlation was a negligible 0.01.  A medium-strength correlation has an r of about .5, and a strong correlation has an r of about .7.  The results discussed above are consistent with the idea that the brain is not the source of the human mind, and is not the storage place of human memories.  Under such an idea, we would expect there to be no strong correlations between brain states and unemotional mental activity such as calm thinking or calm recall.