The site www.neurosciencenews.com is a frequent supplier of dubious brain-related stories, very many of which start out with unfounded headlines not matching anything that was actually observed. The site's latest not-really-true headline is one proclaiming "First Evidence of Replay During Sleep in the Human Motor Cortex, Which Governs Voluntary Movement." As a general rule, you should tend to be suspicious of anyone claiming to provide the first evidence of something, particularly anything having to do with the brain and the mind.
The article attempts to persuade us that while someone was sleeping, his brain was replaying some memory of a motor skill that the person had recently learned. The article refers to a scientific paper that provides no robust evidence of such a thing, providing no justification for its title of "Learned Motor Patterns Are Replayed in Human Motor Cortex during Sleep." The paper is another study guilty of Questionable Research Practices, which are epidemic these days in experimental neuroscience. The paper gives us another example of what is going on very frequently in neuroscience research these days: scientists making claims in titles and abstracts that are not justified by any observations described in the paper.
Here is a quote from the www.neurosciencenews.com article. An extremely dubious speculation is passed off as a "discovery," but while describing this supposed "discovery" the text admits what is going on is "theorizing," something better described as mere speculation.
"Scientists studying laboratory animals long ago discovered a phenomenon known as 'replay' that occurs during sleep, explains neurologist Daniel Rubin, MD, Ph.D., of the MGH Center for Neurotechnology and Neurorecovery, the lead author of the study.
Replay is theorized to be a strategy the brain uses to remember new information. If a mouse is trained to find its way through a maze, monitoring devices can show that a specific pattern of brain cells, or neurons, will light up as it traverses the correct route.
'Then, later on while the animal is sleeping, you can see that those neurons will fire again in that same order,' says Rubin. Scientists believe that this replay of neuronal firing during sleep is how the brain practices newly learned information, which allows a memory to be consolidated—that is, converted from a short-term memory to a long-term one."
No such "replay" was ever discovered. What is going on seems to be simply pareidolia, which is when people eagerly seeking some pattern claim that they have detected such a pattern, like someone checking his toast every day eagerly looking for the face of Jesus, and one day reporting that he finds a piece of toast that looks like Jesus.
Let's imagine some society dedicated to showing that the clouds above us contain the ghosts of dead animals. Given many eager researchers who scan the clouds day after day looking for shapes that look like the shapes of animals, such a society would probably be able to report some successes, finding a few clouds that look like animals. Similarly, let us imagine some experimenters want to show that some brain activity occurring during some motor activity is "replayed" during sleep. Given eight hours of recordings of brain activity during sleep, it will not be too unlikely that such experimenters would report that sometime during sleep there was some brain wave activity that looked like the brain wave activity that occurred when the motor activity occurred when a subject was not asleep.
Below are some of the things that can help you sort out whether or not robust evidence has been provided:
(1) Look for adequate sample size. If a study used 15 or more subjects per study group, it is a good sign that the study may have used an adequate sample size. The "Learned Motor Patterns Are Replayed in Human Motor Cortex during Sleep" paper uses the grand total of only one subject.
(2) Look for an adequate number of control subjects. A well-designed study will use an adequate number of control subjects. We can imagine how control subjects could have been effectively used in a study like this. Brain waves could have been read from 30 subjects, 15 of whom had learned something, and 15 of whom had not learned that thing. But the "Learned Motor Patterns Are Replayed in Human Motor Cortex during Sleep" does not mention any control subjects. All measurements seemed to have occurred from only one subject.
(3) Look for a well-designed blinding protocol. A well-designed study will use a blinding protocol designed to minimize the chance that researchers will observe and analyze data in a biased way to get whatever result they are hoping to get. The "Learned Motor Patterns Are Replayed in Human Motor Cortex during Sleep" paper does not mention any blinding protocol or blinding procedure. If the study had been done properly, analysts would have been blind as to whether brain waves they were analyzing came from the control subjects who had not learned the motor skills that were supposedly "replayed" or from subjects who had learned such skills.
(4) Look for pre-registration. With a pre-registered study, scientists commit themselves to one particular way that data will be gathered and analyzed, a method publicly committed to before any data is gathered. When pre-registration is not used, we should always be suspicious that scientists have simply "sliced and diced" data in as many ways as they wanted, until it coughed up something maybe looking a little like the desired effect. The use of pre-registration minimizes the chance that a scientific paper is a kind of "keep torturing the data until it confesses" affair. The "Learned Motor Patterns Are Replayed in Human Motor Cortex during Sleep" paper is not a pre-registered study.
(5) Look for a statement of an effect size. When robust evidence has been found, researchers will typically report an effect size. The "Learned Motor Patterns Are Replayed in Human Motor Cortex during Sleep" paper does not report any effect size.
We can imagine what a study might look like if it were to show convincing evidence of neural replay during sleep of a learned muscular behavior. We might see one long squiggly line showing a brain wave recorded when the muscular activity was occurring when the subject was awake. We might then see another long squiggly line showing a brain wave recorded at some point during sleep. The two wiggly lines (each with many up crests) might match exactly, in a way that might be unlikely to occur unless the brain was replaying a muscle memory. We would see in the paper an impressive visual showing one long wiggly brain wave line exactly matching another long wiggly brain wave line. The "Learned Motor Patterns Are Replayed in Human Motor Cortex during Sleep" paper has no such visual.
In the section of the paper entitled "Quantification and Statistical Analysis," we have a "jargon gobbledygook" description of the tortuous statistical rigmarole that went on, a section that might have honestly been labeled "Desperately Seeking Replay Evidence." Below is a description of only part of the byzantine "Rube Goldberg machine" statistical maneuvering that was occurring:
"We used these templates to probe for evidence of replay during the resting and sleeping epochs as follows. At each time-step of the neural recording, for each of the two spatial dimensions, we calculated the cross-correlation between the template and the output of the steady-state Kalman filter. This process yielded two time-series of correlation coefficients equal in length to the time series of the neural recording. Separately for the X- and Y-dimensions, the 98th (for Session 1) or 99th (for Session 2) percentile of the correlation coefficients was chosen as the threshold to designate an activity pattern as a template match. We designated instances when the correlation crossed threshold in both the X- and Y-dimensions simultaneously as simultaneous threshold crossing events (STCEs). STCEs occurring over neighboring time-steps are classified as a single event. STCEs occurred tautologically during the awake task performance blocks. When occurring during rest or sleep, we refer to these instances as putative replay events. The specific percentile implemented as the threshold for a session was selected to optimize the performance of STCEs to correctly identify successful target trials and not identify unsuccessful target and all distractor trials during the active task performance. This was quantitatively operationalized by finding the integer percentile that jointly maximized the sensitivity and specificity (i.e., the Youden's J statistic: sensitivity + specificity − 1) of STCEs to accurately identify successful target trials during awake task performance. To evaluate whether there is relative preservation of neuronal firing sequence during these putative replay events, we determined the order of neuronal firing during each successful target trial and each putative replay event by calculating, for each channel (representing the single or multiunit activity recorded), the time bin within 4 s after the onset of the task completion or replay event that had the maximum firing rate. Thus, for each event (task performance or putative replay), a 96-element sequence was identified. To determine the preservation of firing order across events, we calculated the pairwise matching index Im between each task completion and putative replay event, using the approach derived by Ji and Wilson (2007) where Im is defined as follows. For an M-channel recording, there are M(M – 1)/2 pairs of channels; between two events, let m be the number of pairs that have the same order of peak firing between the two events, and n be the number of pairs that have the opposite order. Define Im = (m – n)/(m + n), such that Im is bounded by [−1, 1]. Two events with precisely the same sequence of activation will have Im = 1 and two events with exact opposite order of activation would have Im = −1. To determine whether the distribution of matching indices we observe are greater than would be expected by chance, we generated a control distribution by calculating the matching indices of 100,000 pairs of randomly generated 96-element-long sequences...We randomly selected 100 segments of neural activity, each equal in duration to the successful target sequence templates, from the 30 min period of rest recorded immediately before the task blocks. We used the output of the Kalman filter generated by these randomly selected segments to produce a series of 100 pseudo-templates. For each pseudo-template, we repeated the template-matching procedure described above, calculating the cross-correlation between the template and the Kalman filter output at each time-step of the recording, and counted the number of STCEs for each pair of pseudo-templates. Because the generated distribution was highly left-skewed, we used a Wilcoxon rank-sum test to compare this distribution with our observed outcome from the true target template to determine whether the number of observed putative replay events was greater than would be expected by chance (i.e., compared with the distribution of STCEs produced from the pseudo-templates). Because the random 'pseudo-template' control described above did not necessarily preserve neural firing rate statistics, as a second control, we performed an alternative bootstrap procedure that specifically preserved the statistics of the underlying neural firing. In this control, for each of 100 iterations, the Kalman filter output for the duration of the recording was broken into 5 min segments. Within each segment, we used the discrete Fourier transform to randomize the phase of the X and Y dimension of the Kalman filter output. We then reassembled the segments into full-time-series and performed the same cross-correlation matching procedure described above using the true target templates....To assess for neuronal replay of target trajectories at different speeds, for each recording session, we used cubic splines to fit the pair of target Kalman filter trajectories and then adjusted the duration of the template using a temporal dilation/compression factor we define as τ. We varied τ over 18 values from 0.1 to 10. For each value of τ, the number of STCEs was calculated during each epoch as above. We ran the phase-randomized bootstrapping control at each value of τ to assess for statistical significance."
I can give an analogy for what seems to be going on above. Although I have never used any version of the Photoshop software, I hear that it has many utilities called filters that allow you to make various transformations of images. Imagine if someone kept photographing his toast, but never seemed to get an image of Jesus. He might try playing around with Photoshop filters, subjecting each toast photo to many types of filters, until he finally got something that looked a little like Jesus. That would be rather like what is going on in the "Learned Motor Patterns Are Replayed in Human Motor Cortex during Sleep" paper. Sifting through hours of brain wave recordings of a sleeping subject, the authors seem to have played around with strange statistical manipulations until they got something that they can claim as some evidence of a brain replaying a memory during sleep. It seems like a "keep torturing the data until it confesses" kind of affair.
The "so many zig-zags" statistical procedure described in the "Learned Motor Patterns Are Replayed in Human Motor Cortex during Sleep" paper is so complicated a "hall of mirrors" with so many "madhouse rules" with a "make it up as you go along" kind of stink that no one will ever be able to impressively reproduce it using the same procedure, nor will the authors ever be able to justify the strange arbitrary analysis choices they made (when using an algorithm like a huge vat of tangled spaghetti). "Going deep down the rabbit hole" like that isn't sound experimental science, which generally involves straightforward well-justifiable procedures to yield reproducible results. The authors have not provided any robust evidence at all of brains replaying memories during sleep.
The Neuroscience News site at www.neurosciencenews.com has for very long been guilty of publishing unfounded headlines that are not justified by any research discussed in the story below the headlines. Besides the example discussed above, another example is the not-actually-true headline we recently saw at this site, a headline of "Molecular Mechanisms Behind Learning and Memory Identified." The story refers to a scientific paper that merely dealt with aversive memory in mice. The paper reveals that the study (which failed to follow a blinding protocol) used way-too-small study group sizes such as groups of only 6 or 7 or 8 mice. As a general rule of thumb, 15 subjects per study group is the minimum for a moderately convincing result. It's the same old story that has been going on for decades in the field of experimental neuroscience: experimenters using way-too-small study groups, and getting what are probably only false alarms, with the experimenters wrongly proclaiming that some important discovery was made.The authors of the paper would have discovered how way-too-small their study group sizes were if they had done what should be done by anyone doing an animal experiment: the performance of a sample size calculation, in which you estimate how large a study group size is needed in order for the study to have a good statistical power. The authors confess that they failed to perform such a calculation. They state this:
"The sample sizes were not pre-determined. For all molecular biology experiments, cellular biology experiments, and behavior tests, sample size was chosen according to previous studies."
Since the use of way-too-small study group sizes is currently an epidemic in experimental neuroscience, with most experimenters failing to use adequate sample sizes, you do not at all justify your choice of a sample size by saying "sample size was chosen according to previous studies."
Another example of a recent unjustified headline on www.neurosciencenews.com is a headline of "Brain Region Found to Play a Crucial Role in Weighing Information From Different Sources." No such thing was actually discovered, because the scientific study mentioned was based on analysis of only two monkeys, a study group size way too small for a reliable result.
No comments:
Post a Comment