In my post "How the Academia-Cyberspace-Pharmaceutical-Biotech-Publishing Complex Incentivizes Bad Brain Research" I gave a long analysis of some of the social and economic factors that are driving poor-quality neuroscience and misleading claims in the press about such research. Below is a quote from one of the bullet lists in that post, part of a much larger list of items in that bullet list:
- Once a study has been published in a scientific journal, it is announced to the public by means of a university press release or college press release. Nowadays university press offices are notorious for their hype and exaggeration, and routinely make interesting-sounding claims about new research that are not justified by the observations in the scientific paper being announced. An abundance of such hype helps to glorify universities, keep them in the public's eyes, and helps to justify the very high tuition rates of universities. Misstatements in press releases come both from press office copywriters who are encouraged to glorify and hype new research, but also from scientists quoted in the press releases, who often make unwarranted or false claims about their own research, in an effort to maximize the attention such research will get.
- There is nowadays an extremely large cyberspace industry that has an incentive to uncritically parrot university press releases, and an additional incentive to further exaggerate and hype the results reported by such press releases. The incentive comes in the form of web pages containing ads that generate revenue for the owners of web pages. So we see an enormous "clickbait" effect, in which sensational-sounding headlines appear on web pages, with the headlines being hyperlinks that take the reader to a page filled with ads. Every time a page with such ads is viewed, money is generated for the owner of the web site. The "clickbait" headlines often take you to pages discussing research that never justifies the sensational clickbait headline. For example, you may see on some web page a headline "Scientists Make Giant Leap in Understanding Memory." Clicking on that headline will typically take you to some story about merely another very poorly designed study using mice, something that is no actual progress in understanding memory.
The two paragraphs above were only part of a much larger explanation of why we get these days so many misleading claims appearing in neuroscience papers and press articles about such papers. The paragraphs written in 2023 describe well two recent 2024 cases in which groundless claims of progress were made about neuroscience research. In these cases we got particularly egregious and glaring examples of misleading claims. Great breakthroughs were claimed, even though no evidence was provided to justify such claims.
The first was a January 2, 2024 press release from Dartmouth College entitled "Researchers identify new coding mechanism that transfers information from perception to memory." Nothing of the sort actually occurred. Scientists have no robust evidence that memories are stored in brains, and have no understanding of how information could be transferred in a brain from perception to memory storage. Scientists have no understanding of how something that you see could be stored in the brain as a memory.
The text of the press release makes the untrue claim that "researchers identified a neural coding mechanism that allows the transfer of information back and forth between perceptual regions to memory areas of the brain." Nothing in the corresponding scientific paper justifies such a claim.
The paper is behind a paywall. But doing some additional work, you can find a preprint of the paper ("A retinotopic code structures the interaction between perception and memory systems") on the biology preprint server. The link here gives the full text of the paper. The paper is yet another example of the Questionable Research Practices so predominant in today's experimental neuroscience. The study group sizes are way too small, being no greater than 17 and sometimes as small as only 8 or 9. The study is based on fMRI analysis, but it has been established that fMRI analysis like this is hopelessly unreliable when the study group sizes are so small. See the appendix of this post for a discussion of why very large study group sizes are needed for reliable results in fMRI analysis of this type. The study mentions no use of any control subjects or any meaningful use of experimental controls. We read of analysis of a memory activation task, but no analysis of a control state in which subjects were not remembering. An essential element for credibility in a study like this is the following of a rigorous blinding protocol. But no blinding protocol was followed. The paper is also not a pre-registered study. We have a "see whatever you want to see" situation in which the authors were free to analyze the data in any of endless ways until they could report something they wanted to see. The reported result is almost certainly mere pareidolia on the part of the researchers. No credible evidence has been provided that any kind of neural code has been discovered.
In the "Spatial Sensitivity Analysis" part of the paper, we have an example of what sounds like some statistical skullduggery or sleight-of-hand going on:
"On each iteration (1000 iterations total; randomized PMA pRF order): 1) For every pRF in a PMA, we computed the pairwise Euclidean distance (in x, y, and sigma) to all +pRFs in the paired SPA and found the SPA pRF with the smallest distance that was smaller than the median distance of all possible pRF pairs, 2) we required that all pRF matches were uniquely matched, so if an SPA pRF was the best match for two PMA pRFs, then the second PMA pRF was excluded....
Second, we compared the correlation in trial x trial activation matched (versus non-matched) pairs of pRFs with 'unmatched' pRFs. To create the 'unmatched', random pRF pairings, we randomly sampled pRFs in the memory area (repeated 1000 times). We then computed the unique correlation in trial x trial activation during recall between SPA pRFs and PMA pRFs, using the same procedure as in our main analysis (e.g., the partial correlation between SPA pRFs with PMA -pRFs, controlling for PMA +pRFs) for each iteration of the pRF matching. We compared the mean of the Fisher transformed partial correlation values across the iterations for the matched pRFs with the mean of random (i.e., unmatched) pRFs. To ensure that matched pRFs had better corresponding visual field representations than unmatched pRFs, we calculated the visual field overlap between pRF pairs in the matched samples, compared with the random samples (average dice coefficient of the visual field coverage for all matched versus unmatched iterations)."
This is only a small fraction of the byzantine "spaghetti code" statistical rigmarole that was going on. No one should be impressed by the language above. To some this gobbledygook may sound "scientific" or "mathematical," but what it actually is a confession of some convoluted, arbitrary "Rube Goldberg" analysis pathway that is the opposite of a straightforward analysis. We get a strong smell here of "keep torturing the data until it confesses." The more convoluted an analysis and data manipulation pathway, the less confidence we should have in it. If someone told you that he processed the famous Zapruder film of John Kennedy's assassination by doing a simple straightforward filter, you might be impressed if this yielded something unseen by someone looking at the original film, such as what looked like a gunman hiding in the grassy knoll. But if someone told you that he did a long, convoluted, arbitrary series of manipulations that yielded such an impression of a gunman hiding in the grassy knoll, you should have no confidence in an analysis so arbitrary and byzantine. And when we read scientific papers that describe an arbitrary convoluted spaghetti-code analysis pathway (one of thousands of possible analysis pathways), we should have very little confidence in the claimed results.
We have in the press release quotes by two of the researchers claiming to have found something that the research did not actually find, because the research was such a very bad example of Questionable Research Practices. Often in today's neuroscience press releases we have some copywriter at a university press office making claims about research that were never made by the researchers. But in this case the fault for the misleading press release lies largely with the researchers themselves, who gave press release quotes claiming to have shown something their shoddy methods research never showed.
We have the same thing going on in another January 2024 press release, a press release making the groundless claim "New study reveals how your brain organises experiences in time." The press release makes a groundless claim of a "breakthrough discovery." Nothing mentioned in the press release justifies any such claims. We have a reference to a new scientific paper ("Minute-scale oscillatory sequences in medial entorhinal cortex") that is another very bad example of Questionable Research Practices. The study group sizes were ridiculously small, consisting of only 3 mice and 4 mice. No blinding protocol was used. We read no mention of control subjects. We read that "Power analysis was not used to determine sample sizes." A proper use of sample size calculation (power analysis) would have revealed that the study group sizes were way too small for a reliable result.
As in the previous case, the paper confesses an absurdly convoluted and byzantine analysis pathway that is the opposite of straightforward. Below is only a small fraction of the "spaghetti code" rigmarole gobbledygook that went on:
"First, the similarity in preferred phases of all cells within spatial bins of the FOV was used as a proxy for local gradients. The similarity in preferred phases was calculated as the mean vector length (MVL) of the distribution of preferred phases within each bin of the FOV. The analysis was performed for individual sequences separately. For each of the 15 oscillatory sessions (over 5 mice), the FOV was divided into spatial bins of 100 μm x 100 μm (6 × 6 bins in 10 sessions, 10 × 10 bins in 5 sessions), or 200 μm x 200 μm (3 × 3 bins in 10 sessions, 5 × 5 bins in 5 sessions) (note that for 10 of the 15 oscillatory sessions the FOV was 600 μm x 600 μm, mice no. 60355, no. 60584, no. 60585; while for 5 of the 15 oscillatory sessions the FOV was 1,000 μm × 1,000 μm, mouse no. 59914; mouse no. 59911 did not show the oscillatory sequences). Next, the preferred phase of each cell per sequence was calculated (as we did in ‘Correlation between differences in preferred phase and anatomical distance’) and for each sequence and every spatial bin of the FOV the MVL was computed (only spatial bins with 10 or more cells were considered). If the MVL was 0, then all preferred phases in that bin were different and homogeneously distributed between −π and π, whereas if the MVL was 1 then all preferred phases were the same. In the presence of a travelling wave, each bin should have a high MVL value compared to chance levels. Statistical significance was determined by repeating the same MVL calculation after shuffling the cells’ preferred phases within the FOV 200 times, and using, for each spatial bin, a cutoff for significant of 95th percentile of the shuffled distribution...To determine whether long sequences act as a template for the formation of given activity patterns in a neural population, we built a simple perceptron model in which 500 units were connected to an output unit (Extended Data Fig. 12a). There was a total of 500 weights in the network, one per input unit. The total simulation time was 120 s, with 3,588 simulation steps and a time step of 33.44 ms (original time step was 129 ms, to mimic the bin size used in calcium data, rescaled so that the length of one of the input sequences was 120 s, similar to the length of the sequences in Fig. 2b). The response of the output unit was given by R = WX, where W was the vector of weights, and X the matrix of input activity (each column is a time step, each row is the activity of one input unit). The weights were trained such that the output unit performed one of two target responses (see below). For each target, we trained the model using as input periodic sequences with 5 different lengths (one length per training), covering the range from very slow to very fast as compared to the characteristic time scale of the targets (100 s).''
This smells like the same situation as in the previous paper: keep torturing the data until it confesses, or until you get the faintest whisper you can call a confession.
No research was done to show that the brain "organizes memories in time." Since the research involved only mice rather than humans, the claim that the research "reveals how your brain organizes in time" is a particularly egregious misstatement.
Convoluted "spaghetti code" analysis pathways are only one of many ways in which scientists can conjure up phantasms that don't exist. To read about other such ways, read my post "Scientists Have a Hundred Ways To Conjure Up Phantasms."
You can search for the presence of arbitrary rigmarole "spaghetti code" analysis pathways in a scientific paper by manually checking, or by searching for the word "iterations." The word "iteration" will typically indicate that the data was passed through processing loops of a computer program. Once data is being passed through such processing loops, it is pretty much an "anything goes" situation in which the scientist may play around with the data according to whatever arbitrary whim he had when writing some programming code. I know from my many years in software development that professional computer programmers often commit programming errors. I would imagine that scientists dabbling in computer programming probably commit far more programming errors than professional software developers. Or, the code may be written by some software developer who does not well understand the data. When scientific experiments rely on programming code (as in the two cases above), almost always the code is not published with the paper. Or, if some promise is made about code accessibility, it is typically something where someone wanting to see the code will have to "jump through hoops" to obtain it. Often there are empty promises that the code is "available upon reasonable request," with such requests not producing responses, or empty-sounding promises such as the promise that the code "will be published on GitHub."
The "Minute-scale oscillatory sequences in medial entorhinal cortex" paper mentioned above does actually publish its code, and you can view it using the link here. Opening up one of the many programming code files by using this link I see code that looks like the worst kind of gobbledygook rigmarole spaghetti code. You might describe what is occurring as "keep torturing the data with bizarre arbitrary-looking programming loops until it whispers the faintest confession." Looking at other programming files in the repository, I see that quite a few other programming files have similar-looking code, which makes you think: "What on Earth were these guys thinking?" When code is as sparsely commented as this code, and has so many arbitrary-looking inscrutable lines, it is usually true that no one understands what the code is doing except the original programmer; and it usually also is true that even the original programmer does not know what exactly the code is doing. What you have is a goofy black box doing God-only-knows-what kind of distortions and transformations of the original data. When did our neuroscientists start thinking they had a license to play with their data in a hundred obscure murky convoluted ways rather than just analyzing the original data in a straightforward way? The situation reminds you of people who think it is okay to apply unlimited photo filters to their smartphone snaps so that their obese old bodies standing in front of average houses look like slim young bodies standing in front of mansions.
When neuroscience research degrades into manipulation malarkey and contortion convolutions, in an "anything goes" fashion in which people are not thinking "stick to the original data gathered from instruments," it is a very sad state of affairs.
How often is it like this?
Appendix: A press release from the University of Minnesota Twin Cities announces results which indicate that such small-sample correlation-seeking brain imaging experiments are utterly unreliable. The headline of the press release is "Brain studies show thousands of participants are needed for accurate results."
We read this:
"Scientists rely on brain-wide association studies to measure brain structure and function—using MRI brain scans—and link them to complex characteristics such as personality, behavior, cognition, neurological conditions and mental illness. New research published March 16, 2022 in Nature from the University of Minnesota and Washington University School of Medicine in St. Louis...shows that most published brain-wide association studies are performed with too few participants to yield reliable findings."
The abstract of the paper in the science journal Nature can be read here. The paper is entitled, "Reproducible brain-wide association studies require thousands of individuals."
The press release tells us this:
"The study used publicly available data sets—involving a total of nearly 50,000 participants—to analyze a range of sample sizes and found:
- Brain-wide association studies need thousands of individuals to achieve higher reproducibility. Typical brain-wide association studies enroll just a few dozen people.
- So-called 'underpowered' studies are susceptible to uncovering strong but misleading associations by chance while missing real but weaker associations.
- Routinely underpowered brain-wide association studies result in a surplus of strong yet irreproducible findings."
No comments:
Post a Comment