Be Skeptical of "Consciousness in Comatose" Claims, Which Are Probably More Brain Expert Pareidolia

Neuroscientists often make groundless boasts of having discovered things in brains that are not actually there. What is going on is pareidolia, people reporting seeing something that is not there, after wishfully analyzing large amounts of ambiguous and hazy data. It's like someone eagerly analyzing his toast every day for years, looking for something that looks like the face of Jesus, and eventually reporting he saw something that looked to him like the face of Jesus.  It's also like someone eager scanning the clouds, looking for animal shapes, while trying to confirm his belief that after an animal dies, his ghost goes up and lives in the clouds. 

On this blog I have provided very many examples of neuroscientist pareidolia. Some examples can be found in my posts here: 

Neuroscientists Claim "Drifting Representations," But It's Mainly Just Their Pareidolia

Pareidolia Progress: Neuroscientists Get Better and Better at Seeing Things in the Brain That Are Not There

Pareidolia Helps Neuroscientists Getting Nowhere Trying to Show a Brain Basis for Memory

Some Brain Wave Analysts Are Like "Face of Jesus in My Toast" Claimants

Normally when neuroscientist pareidolia occurs it is relatively harmless. There is the epistemic harm and intellectual harm caused when bad evidence is provided of things that are not true, but little other harm. However, one type of neuroscientist pareidolia can cause bad psychological harm in the relatives of those who are in a coma. I speak of the case of neuroscientists making shoddy claims of having discovered consciousness in comatose patients. 

We read some examples in a Harvard Medicine article entitled "The Covert Consciousness Dilemma." The article is written by someone falling "hook, line and sinker" for some very dubious claims that were never well-supported.  Early in the article the writer states this:

"Around the same time, Edlow heard about a paper published in Science that stunned him. The 2006 study focused on a patient who sustained extensive brain injuries following a traffic accident. Six months later she was still in a vegetative state. She showed no evidence of purposeful behaviors, such as following commands or making intentional movements. But then the researchers put her in an fMRI machine. While scanning her brain, they asked her to imagine walking through her house, moving slowly from room to room. And they told her to imagine playing a game of tennis, swinging for the ball with forehand and backhand strokes... Her brain responded to the commands with activity patterns that matched those of a healthy person, as if to say, I’m still here."

The 2006 study referred is the very low-quality paper "Detecting Awareness in the Vegetative State," which fails to provide any decent evidence justifying its title. The paper provides no robust evidence that any consciousness was detected in the single subject it studied. The paper claims that "Speech-specific activity was observed bilaterally in the middle and superior temporal gyri, equivalent to that observed in healthy volunteers listening to the same stimuli (fig. S1)." The Figure S1 referred to is one of those extremely misleading visuals in which neuroscientists artificially color-highlight particular regions of the brain that showed only tiny differences such as 1 part in 200, to create the false impression that there was some big difference in brain activity in some region which merely had a difference such as 1 part in 200. 

Here is the Figure1 of the paper 

This visual is not a visual produced directly from any brain scan. The visual was produced through a process of artificial construction designed to make tiny differences such as 1 part in 200 look like big differences. Such methods are misleading. A correct visual showing the brain activity would show differences that cannot even be recognized by a human. 

The authors of the paper fail to caption their Figure 1 correctly. They say that it shows what occurred during "tennis imagery in the patient," even though they have no evidence that any tennis imagery was occurring in the mind or brain of their comatose patient when this scan was taken. Looking closely at Figure 1 above, we fail to even see any close match. The highlighted regions in the first row look significantly different from the highlighted regions in the second row. No good evidence has been provided that any such thing as "tennis imagery" or consciousness was occurring in the patient. 

There is actually no evidence in the literature of neuroscience that there exists any such thing as a neural correlate of imagination. If you scan someone's brain while asking him to keep a blank state of mind, and compare the scans to scans taken while the person was asked to imagine something, you will see no difference if the person kept perfectly still. There will be variations here and there such as 1 part in 200, but such variations can be explained as mere random variations, rather than something that resulted from an act of imagination. Muscle movements do produce blips in EEG readings and fMRI scans, but such blips are not evidence of thinking. 

The paper "Neural substrates of envisioning the future" describes a study looking for neural correlates of imagination. 21 subjects were brain scanned while they did three tasks: remembering an event in their past (such as a birthday in the past), imagining an event in their future (such as a future birthday they would have), or imagining an event in someone else future.  Figure 1 shows unimpressive results, with percent signal changes no greater than about 1 part in 200, which are the kind of differences one would expect to get from random fluctuations, even if people don't use their brains for imagining or remembering.  We read of a "statistically indistinguishable pattern of activity across time while subjects envisioned their personal future (SF) and recollected the past (SR) in response to a series of event cues (e.g., Birthday)."  The results when people were "imagining a future individual in similar scenarios" also fail to show any clear sign of neural involvement, with percent signal changes no greater than about 1 part in 200.  

 The low-quality paper "Detecting Awareness in the Vegetative State" fails to provide any decent evidence for consciousness in the comatose. To have decent evidence for such a thing, you would need a much larger sample size than the paper's way-too-small sample size of a single subject. The authors are guilty of runaway pareidolia when they claim this about their comatose patient "Moreover, her decision to cooperate with the authors by imagining particular tasks when asked to do so represents a clear act of intention, which confirmed beyond any doubt that she was consciously aware of herself and her surroundings." They have provided no decent evidence of either imagination, intention, cooperation or consciousness in the comatose patient they brain scanned.  The authors engage in the old fallacy of claiming that "significant activity was observed" in particular brain regions when the subject was asked to imagine something. All brain regions are continually active in any living patient with a heartbeat, so showing that "significant activity was observed" in particular brain regions does nothing to show consciousness, intention or imagination. 

We then have in the Harvard Medicine article an emotional quote from neurologist Brain Edlow saying that this low-quality paper "just blew my mind," followed by the groundless claim that this paper "completely reframed the way that we think about consciousness and opened up the possibility that a substantial number of patients may be conscious even if we cannot detect purposeful behavior at the bedside." Once again, we have a brain expert getting all super-excited over the flimsiest of evidence. 

The Harvard Medical article then refers us to a 2024 paper with a larger size, the paper "Cognitive Motor Dissociation in Disorders of Consciousness." The paper is one of those deals in which you have a claim of a fairly large sample size, but the claim is made by adding up the study group sizes in a bunch of papers, which all used very small samples. 

The two sentence abstract of the paper starts out by making unfounded claims, saying this:

"Patients with brain injury who are unresponsive to commands may perform cognitive tasks that are detected on functional magnetic resonance imaging (fMRI) and electroencephalography (EEG). This phenomenon, known as cognitive motor dissociation, has not been systematically studied in a large cohort of persons with disorders of consciousness."

No such phenomenon has ever been established by high-quality studies, and this paper is not any such high-quality study. The "detection" it refers to are very likely example of pareidolia or errors-to-be-expected, in which over-eager researchers see things that are really there, because of a biased interpretation of borderline, varying data in which an over-eager analyst could see 101 things that are not really there. 

The authors claim, "We detected cognitive motor dissociation in 60 of the 241 participants (25%) without an observable response to commands, of whom 11 had been assessed with the use of fMRI only, 13 with the use of EEG only, and 36 with the use of both techniques."  The visual below shows the type of devices used. 

The paper incorrectly states, "Cognitive motor dissociation is an established phenomenon in which persons with severe brain injury who are behaviorally unresponsive to commands show brain activation on functional magnetic resonance imaging (fMRI) or electroencephalography (EEG) when presented with cognitive tasks, such as motor imagery commands." No robust evidence has ever shown the existence of such a "cognitive motor  dissociation" phenomenon. 

Supplementary Figure S4 (in the Supplementary Information of the paper) gives us an indication of how weak the evidence of this paper is. The figure refers to a CRS-R scale that is a measure of how deeply someone is in a coma. According to an AI overview, "A total score of ≥ 8 generally indicates a transition from a vegetative state to a minimally conscious state, while a score of ≥ 10 helps identify patients who have achieved emergence from a minimally conscious state." The Figure S4 graph below shows no robust evidence of  any responses from any patients with a score of less than 11 on this CRS-R score. Below is the graph:

The left half of the graph plots responses in those being in stronger and deeper comas. The right half of the graph shows responses in those who are not fully comatose.  The claimed evidence for responses (as detected by fMRI or EEG activity) is found in the bar sections that are light blue or pink. 

But as you can see by closely studying this graph, there is almost no pink or light blue color in any of the bars on the left half of the graph. Any of the very small pink areas we see in the left half of the graph above can easily be explained as being mere pareidolia or garden-variety analysis error from over-eager researchers eager to report "responses" after analyzing data that did not actually show any good evidence of a response. 

So we see the "secret sauce" behind this paper "Cognitive Motor Dissociation in Disorders of Consciousness." It may have some got some evidence of responses from the subjects that underwent scanning by fMRI scanning or EEG scanning. But almost all of those claimed responses come from those not fully comatose (those scoring higher than 10 on the CRS-R scale). No strong evidence has been produced of responses or cognition or imagination or will in any of those who are fully comatose (the people graphed on the left half of the graph above). 

Normally the misleading language and "see what you are hoping to see" pareidolia of brain experts is fairly harmless. But in this case such pareidolia is extremely harmful. The people doing low-quality studies like this are causing the relatives of those in a coma to needlessly fear that their relatives are enduring what many would regard as a fate worse than death: a fate of being conscious in a body when you cannot open your eyes and cannot move your muscles.  There is no robust evidence that anyone in a coma is suffering such a fate. "Eager for another paper publication" brain experts are causing needless mental anguish in the relatives of those in comas, by groundlessly causing them to fear their relatives are undergoing some horrible fate, when there is no sound evidence basis for such fears.  When studying such papers, we should always remember that brain experts studying the ever-varying data from fMRI scans and EEG readings have a long history of frequently claiming to see things that are not really there, and a long history of conjuring up phantasms that don't really exist. 

You cannot actually tell whether someone is imagining something by looking at EEG readings taken from such a person's brain or by looking at fMRI scans. Studies that claim to provide evidence for such a thing are often studies that improperly leverage EEG traces of muscle movements. If you give someone an instruction such as "imagine yourself playing tennis," a person might well make subtle movements matching such thoughts, movements that show up in EEG readings. Evidence of muscle movements are not evidence of imagination. No one disputes that muscle movements produce EEG blips. 

A look at one of the papers referenced by the "Cognitive Motor Dissociation in Disorders of Consciousness" paper shows the kind of arbitrary and convoluted statistical rigamarole that was going on to gin up some of these claims of "responses" in comatose patients being spoken to:

"We used two complementary methods to determine the significance of differences in the frequency content of the EEG signal between the task and rest conditions: a univariate (frequency-by-frequency) approach and a multivariate approach. For each subject, both analyses were applied on a channel-by-channel basis to each run individually and to all runs combined.

For the univariate approach, we used a z-statistic, the Two Group Test (TGT) (Bokil et al., 2007), as implemented by the Chronux toolbox routine, two_group_test_spectrum (http://chronux.org), with a cutoff of p≤0.05 by jackknife method. Because spectral estimates within 2 Hz of each other are correlated by the taper functions, a difference identified by the TGT was only considered significant if it was present for all frequencies contiguously over a range greater than 2 Hz. This implies significance over at least two neighboring but non-overlapping windows of the multi-taper estimate and is indicated in figures with a rectangle drawn around the results (Figure 3B). Spectral differences over ranges narrower than 2 Hz represent only a trend to significance. To compensate for multiple comparisons (60 frequencies per channel in 29 or 37 channels), the False Discovery Rate (FDR) (Benjamini and Hochberg, 1995; Benjamini and Yekutieli, 2001) was applied to the TGT p-values determined from analyses of all runs combined.

To look for spectral differences that might only be apparent if combinations of frequencies are considered, we employed a multivariate approach, Fisher’s linear discriminant (FLD) (Fisher, 1936). This approach has been used successfully for classification of EEG responses to motor imagery (Hung et al., 2005; Bai et al., 2007). To limit dimensionality, we binned the log spectra from 4 to 24 Hz into 2 Hz windows, reducing the spectrum to 10 values. The FLD was then defined as the linear combination of these quantities that maximized the ratio of the power variance between the conditions to the power variance within the conditions. To determine the significance of the FLD, we used a shuffle method: we recomputed the FLD from 1000 shuffles of the two conditions, and determined the p-value as the fraction of shuffled datasets that yielded an equal or larger value to the actual FLD. To take into account the possibility that neighboring snippets had similar spectra because of a slowly changing underlying brain state (rather than the task) (Menzer et al., 2010), the shuffled datasets kept the snippets from the nine-second-response period after each command together during all shuffles. To control for multiple comparisons (since the FLD was applied separately to each channel), the FLD p-value was only considered significant for a channel if it was less than an FDR-corrected rate of 0.05. This is shown as an asterisk on the summary figures (e.g. channel Oz in Figure 3B). For each subject, this analysis was applied to each run individually, and to all runs combined."

Of course, when researchers are free to use "keep torturing the data until it confesses" methods like these (methods as arbitrary, convoluted and opaque as the researchers may wish), it is hardly a surprise that there might be conjured up here and there a little so-called "response evidence," even when data from the brains of utterly unconscious coma patients is being analyzed. 

So many ways to conjure up phantasms that don't exist

Pareidolia Progress: Neuroscientists Get Better and Better at Seeing Things in the Brain That Are Not There

 Our neuroscientists are getting nowhere in trying to show that there is a neural basis for human memory or human thinking. But they have something they can rely on to help hide their lack of progress:  pareidolia. Pareidolia is when you see patterns that aren't really there, like some guy examining his toast every day for years, and then one day saying, "I finally see the face of Jesus in my toast."  A scientist conjuring up some pareidolia can make a nice-sounding progress report when no real progress has been made. I describe some examples of this in my post "Scientists Have a Hundred Ways to Conjure Up Phantasms That Don't Exist." 

There are several factors driving an increase in the ability of neuroscientists to make these pareidolia reports of seeing things in a brain that are not there. They include the following:

(1) Technological improvements have made it more and more easy for neuroscientists to use microelectrodes to record activity from individual neurons. This type of invasive brain intervention yields much more data than you get from the non-invasive technique of reading brain waves by having someone wear an EEG cap with electrodes. Neurons fire randomly at a rate between about 1 time per second and about 300 times per second. Being able to get data from all the individual firings of a large set of neurons randomly firing is almost the perfect seed bed for pareidolia.  The more random, rapidly changing data you get, the easier it is to do noise-mining pareidolia in which you seek for patterns you are eagerly hoping to find. 

(2) Advances in computer programming and AI make it easier-than-ever to create computer programs that manipulate gathered brain data in arbitrary ways. Easy-to-use languages such as Python make it easier for neuroscientists who are not professional programmers to write such programs. AI tools such as ChatGPT allow the creation of artificially generated code that can be used as part of such programs. The easier it is to produce such computer programs, the easier it is to do the kind of "keep torturing the data until it confesses" work that is often the basis of pareidolia, or a pillar of pareidolia. 

(3) Advances in so-called artificial intelligence (AI) and the public accessibility of such techniques make it easier than ever to do noise mining or data mining to extract or construct patterns or claimed patterns that a human might night never be able to find or that humans would never say existed if the humans made a simple, straightforward examination of the data. 

We have a recent example of neuroscientists seeing things that are not really there is the paper "Dynamic coding and sequential integration of multiple reward attributes by primate amygdala neurons" which you can read here. We have two neuroscientists claiming to have found that "neurons frequently signalled reward probability in an abstract, stimulus-independent code." They provide no robust evidence for this claim. To qualify as decent evidence, a study like theirs would have required a study group size of at least 15 subjects. The study group size they used was the way-too-small size of only two subjects, both monkeys. 

Below from Figure 4 of the paper is some of the data gathered from monitoring several hundred neurons of the two monkeys, as they were either resting or presented with some kind of cue. 

The little blips are neuron firing or neuron firing spikes going on while these cues were presented. A human looking at this data will see no pattern. But the authors played around with what we might charitably call "analysis techniques," trying to find some way to squeeze some evidence of a pattern or a code out of this neural noise. In the paper these are called "general linear models" and are given names of GLM 1, GLM 2, GLM 3, GLM 4 and GLM 5. Describing each of these "models," the authors groundlessly claimed that each of them identified a particular neuron encoding in some different way. The claims are never justified, and seem like pure imagination. 

Procedures like this almost invariably involve computer programming code that passes the data through programming loops. It's kind of like what's depicted in the visuals below. 

Normally you get an idea of how objectionable the data processing was by looking at the programming code used by the scientists. There is no excuse for any study of this failing to publish its code, as there are nowadays various platforms and web sites that make it very easy for someone to make their code publicly available online.  Typically in affairs such as this you can examine the programming code, and find some convoluted poorly documented mess that will demonstrate that the data was being monkeyed with in wild and weird ways. If, on the other hand, you were to find some clean, straightforward, well-documented code, it might be a sign of a respectable methodology. 

But often the scientists will not make their programming code publicly available. This is a strong reason for suspecting that the programming code involved is something that the scientists were embarrassed by, some programming horror they were too embarrassed to publish.  In the case of the paper discussed here ("Dynamic coding and sequential integration of multiple reward attributes by primate amygdala neurons" which you can read here), the authors have failed to make their programming code public.  We should have no confidence in any of the author's claims to have found "probability-coding neurons." but we can assume that the authors were not proud of their own coding. 

The authors are just engaging in the most groundless "Jesus in my toast" pareidolia when they make these claims:

"Amygdala neurons frequently signalled reward probability in an abstract, stimulus-independent code that generalized across cue formats. While some probability-coding neurons were insensitive to magnitude, signalling ‘pure’ probability rather than value, many neurons showed biphasic responses that signalled probability and magnitude in a dynamic (temporally-patterned) and flexible (reversible) value code. Specific neurons integrated these reward attributes into risk signals that quantified the uncertainty of expected rewards, distinct from value."

Making these claims the authors are like someone staring at 1000 photos of clouds in the sky, and finally declaring that he sees angel castles and animal ghosts. Among their procedural sins:

(1) They have not done a study with even one seventh of the study group size needed for a study like this to be taken seriously. 

(2) They have not revealed the exact details of their procedural method, because they failed to publish the programming code that their claims are dependent on. 

(3) Instead of testing a single hypothesis and declaring that it either succeeded or failed, they kept playing around with different ways of analyzing data until they found something they thought produced a publishable result. 

All claims that the authors make about finding neurons "encoding" something are spurious and groundless. Their description of the analysis algorithms they fooled around with is a description of a  tangled, tortured gobbledygook rigamarole, some "witches brew" of statistical spaghetti code shenanigans that the authors give no justification for. You could give the data they recorded to 100 neuroscientists and ask them whether they saw anything in the data; and not one of them would claim that they saw any evidence of "probability encoding neurons" unless you primed the pump by mentioning such a notion to them. 

Programming Gone Astray: Iteration Inanity of the Neuroscientists' Distortion Loops

 Quanta Magazine is a widely-read online magazine with slick graphics. On topics of science the magazine again and again is guilty of the most glaring failures. Quanta Magazine often assigns its online articles about great biology mysteries (involving riddles a thousand miles over the heads of PhDs) to writers who lack even a bachelor's degree in biology. Often it will assign such articles to be written by people identified as "writing interns."  The articles at Quanta Magazine often contain misleading prose, groundless boasts or the most glaring falsehoods. I discuss some examples of such poor journalism in my posts here and here and here and here. 

The writers at Quanta Magazine are very often guilty of bootlicking, a word meaning excessive deference to an authority or a superior.  The latest example of bootlicking at the magazine is an article entitled "How 'Event Scripts’ Structure Our Personal Memories." The subtitle makes this very untrue claim: "By screening films in a brain scanner, neuroscientists discovered a rich library of neural scripts — from a trip through an airport to a marriage proposal — that form scaffolds for memories of our experience."  The claim has no basis in fact. The article follows it with this equally untrue claim: " 'Event scripts' are distinct neural fingerprints that encode repeated sequences of events, such those that unfold during a trip through the airport."  No such things have been found. 

The article begins telling us tall tales about neuroscientist Christopher Baldassano, incorrectly stating this: "Then, in 2018, Baldassano found it: neural fingerprints of narrative experience, derived from brain scans, that replay sequentially during standard life events. " No such thing happened. The article is referring to a very low-quality paper co-authored by Baldassano, one entitled "Representation of Real-World Event Schemas during Narrative Perception." 

The study had the following flaws:

(1) The study group sizes in this task-based fMRI study were skimpy, consisting of only 15 or 16 subjects per study group. Referring to study group sizes twice as large, an article on neursosciencenews.com states this: "A new analysis reveals that task-based fMRI experiments involving typical sample sizes of about 30 participants are only modestly replicable. This means that independent efforts to repeat the experiments are as likely to challenge as to confirm the original results."

(2) No blinding protocol was used. 

(3) The paper was not preregistered, and did not test any hypothesis formulated before gathering data, using a method specified before gathering data. 

(4) The paper is a bad example of "keep torturing the data until it confesses" methodology.  The paper has graphs that are not based on simple brain scans, but are instead based on brain scan data after it has been manipulated through the most convoluted pathway of arbitrary contortions. 

Below from the paper is a discussion of only a small fraction of the "keep torturing the data until it confesses" nonsense that was occurring:

"For each story, four regressors were created to model the response to the four schematic events, along with an additional nuisance regressor to model the initial countdown video. These were created by taking the blocks of time corresponding to these five segments and then convolving with the HRF from AFNI (Cox, 1996). A separate linear regression was performed to fit the average response of each group (in the 100-dimensional SRM space) using the regressors, resulting in a 100-dimensional pattern of coefficients for each event of each story in each group. For every pair of stories, the pattern vectors for each of their corresponding events were correlated across groups (event 1 from Group 1 with event 1 from Group 2, event 2 from Group 1 with event 2 from Group 2, etc., as shown in Fig. 2a) and the four resulting correlations were averaged. This yielded a 16 X 16 matrix of across-group story event similarity. To ensure robustness, the whole process was repeated for 10 random splits of the 31 subjects, and the resulting similarity matrices were averaged across splits...To explore the dimensionality of the schematic patterns, we reran the analysis after preprocessing the data with a range of different SRM dimensions, from 2 to 100. The resulting curve of z values versus dimensionality for each region was then smoothed with the LOWESS (Locally Weighted Scatterplot Smoothing) algorithm implemented in the statsmodels python package (using the default parameters). To generate the searchlight map, a z value was computed for each vertex as the average of the z values from all searchlights that included that vertex. The map of z values was then converted into map of q values using the same false discovery rate correction that is used in AFNI (Cox, 1996)....The resampled data (time courses on the left and right hemispheres, and in the subcortical volume) were then read by a custom python script, which implemented the following preprocessing steps: removal of nuisance regressors (the 6 degrees of freedom motion correction estimates, and low-order Legendre drift polynomials up to order [1  duration/150] as in Analysis of Functional NeuroImages [AFNI]) (Cox, 1996), z scoring each run to have zero mean and SD of 1, and dividing the runs into the portions corresponding to each stimulus. All subsequent analyses, described below, were performed using custom python scripts and the Brain Imaging Analysis Kit (http://brainiak. org/)."

This is only a small fraction of the contortion inanity that was going on. The paper has many other paragraphs sounding like the one just quoted.  To see the ugliness of the manipulation muddle that was occurring, you must look at the programming code. The authors have made their code public, and you can see it using the link here.  Looking at their programming scripts, we see an appalling example of arbitrary, unjustifiable  algorithms, the most convoluted spaghetti code.  The brain scan data is being passed through many types of poorly documented programming loops that are doing God-only-knows-what kind of mystifying manipulation. Below is only a tiny part of the bizarre manipulations that were going on.

You might call this "iteration inanity." The output is some kind of utterly artificial "witches' brew" that cannot be called the original data gathered or anything like the original data gathered. We should not have any confidence in any of the main graphs in the paper, because they are all produced by passing brain scan data through spaghetti code convolution contortions similar to the one shown above.  This is a severe example of "keep torturing the data until it confesses," what we might call a Spanish Inquisition level of torturing.  We have some utterly artificial transmogrification mess that is the result of obscure arbitrary  programming manipulations, some gobbledygook rigmarole. The authors have not found any "event scripts" or patterns in the brain.  The only thing they have found is something they have created themselves by spaghetti-code programming that distorts and manipulates the original brain scan data. 

Was this how the mess arose?

When good programmers are writing straightforward programming code and they know what they are doing, they tend to use intelligible variable names such as ThisYearsAccruedInterest or TotalAccruedInterest.  Bad programmers use unintelligible variable names such as "d" or "ev" or "cc" or "np," like in the example above, without any comments documenting the variable names, often because they don't even understand what the variables correspond to, and cannot give an intelligible name corresponding to the variable.  As a general rule, we should tend to distrust any scientific programming that uses undocumented variable names of one or two letters such as "d" or "ev" or "cc" or "np," because the use of such cryptic variable names is a strong reason for suspecting that incompetent programmers are at work. 

The Quanta Magazine article then has a link to another paper by Baldassano and others, one entitled "Top-down attention shifts behavioral and neural event boundaries in narratives with overlapping event script."  The paper relies on the same kind of iterative inanity as the previously mentioned paper. We see the same type of loony-looking loops that make all kind of weird, arbitrary transfigurations and contortions and manipulations of the original data, with only the scarcest comments in the source code to explain what is being done. It's another big heap of spaghetti code nonsense doing God-only-knows-what to the original data.  You can see the manipulation mess by looking at the Python files here. 

Nothing real about the brain is being revealed here. If any "scripts" or patterns were discovered, the authors were merely discovering the outputs of their own data-manipulating programming loops. To claim the output of such distortion loops as being something in the brain is as misleading as picking up 100 stones from the seashore,  forming them into a sculpture of a cat, and then claiming that the waves produced a sculpture of a cat. 

It is rather obvious that our Quanta Magazine writer has not learned how to distinguish good neuroscience research from very bad neuroscience research. That writer states this:

"In 2004, the neuroscientist Uri Hasson and his colleagues at the Weizmann Institute of Science in Israel started carving a path through the thicket of voxels. In one of their studies, five people, while lying in a brain scanner, watched 30 minutes of The Good, the Bad and the Ugly (1966), a spaghetti western starring Clint Eastwood. Comparing the data from the five participants, the researchers noted when and where brain activity surged or waned in unison."

Why would anyone even bother to mention a research study using so obviously too-small a study group size of only five subjects? The writer then gives us one more bum steer. We are given false claims about a study by a neuroscientist named Chen:

"In 2012, Chen joined Hasson’s lab, then at Princeton, and extended the approach to memory. She had people watch the first episode of the television show Sherlock (2010), featuring Benedict Cumberbatch as a modern take on the legendary detective. Then the study participants talked through their memory of it, while still lying in the scanner. The experiment worked. Chen and her colleagues were able to match brain activity recorded during participants’ recollections to specific scenes around 60 seconds long — for example, when Sherlock meets Watson."

The claim is false, because the study was some very low-quality work. The link given in the Quanta Magazine article is to the paper "Shared memories reveal shared structure in neural activity across individuals" which you can read here. The study had the following defects:

• The study used too-small study groups such as one with only 8 subjects and another with only 9 subjects. The authors confess, "No statistical methods were used to pre-determine sample sizes but our sample sizes are similar to those reported in previous publications." It is well-known that neuroscience experiments typically use way too few subjects for results with good statistical power, so you do not have a good excuse for failing to do a sample size calculation (to determine a good study group size) by appealing to other experimenters using study group sizes like yours. 
• The study failed to use a blinding protocol.  The authors confess, "Data collection and analysis were not performed blind to the conditions of the experiments." 
• Instead of simply using the original brain scan data, the authors performed very many obscure and arbitrary convolutions, contortions and distortions of the original data. 

A very long part of the paper describes all the weird data manipulations and convoluted contortions that were occurring. Here is only a very small fraction of that part:

"We performed a resampling analysis wherein the individual participant correlation values for recall-recall and movie-recall were randomly swapped between conditions to produce two surrogate groups of 17 members each, i.e., each surrogate group contained one value from each of the 17 original participants, but the values were randomly selected to be from the recall-recall comparison or from the between participant movie-recall comparison. These two surrogate groups were compared using a t-test, and the procedure was repeated 100,000 times to produce a null distribution of t values. The veridical t-value was compared to the null distribution to produce a p-value for every voxel. The test was performed for every voxel that showed either significant recall-recall similarity (Fig. 3B) or significant between-participant movie-recall similarity (Fig. 3B), corrected for multiple comparisons across the entire brain using an FDR threshold of q = 0.05 (see Methods: Pattern similarity analyses); voxels p < 0.05 (one-tailed) are plotted on the brain (Fig. 4A)"

The authors have not provided a link to their source code. Based on the descriptions of their methods, we may assume that they were using the same kind of unjustifiable distortion loops that go on in the papers of Baldassano.  People with the worst programming are the least likely to make the code public. The Chen paper "Shared memories reveal shared structure in neural activity across individuals" is low-quality work that fails to follow good standards of research. No robust evidence has been provided of "shared structure in neural activity" when the same memories are experienced.  The authors seem to have merely discovered something they created themselves through their strange contortions and manipulations of data. 

The Quanta Magazine article is a very bad example of bootlicking. We have all kinds of claims that scientists accomplished something, when most of these things were not actually done, because the methods used were so poor.  Instead of such fanboy swooning, the author should have put the methods of the discussed neuroscientists under stringent critical scrutiny, which would have mainly revealed the defective methods being used. 

Part of the problem with studies like this is that we do not get any chronological account of the different attempts at fooling around with the brain scan data that was produced. We only get a result of some final algorithmic result that the authors had, after a long process of "keep torturing the data until it confesses in the weakest whisper." We may presume that what often goes on is something rather like this:

Programmer: Well, that ends my 18th programming attempt to squeeze some "patterns" out of this brain scan data, and I still have nothing. I'm getting nowhere. It's like trying to squeeze blood from a stone. 

Scientist: Keep trying! Be more creative!  Add, pad; slice, dice;  merge, purge; mix, fix; ruffle, shuffle; sift, shift; combine, align; inflate, conflate; shrink, link and sync; crop, drop, and swap; bend, blend, rend and mend; ditch, hitch, stitch and switch. Try every kind of programming loop you can think of, to try to gin up something from this data that we can call a pattern, or something we can claim as a possible representation. 

Programmer: Do I have to save all the earlier versions of my code that failed?

Scientist: Hell no. We only describe the FINAL version of the code in our paper. 

Neuroscientists Senselessly Think They Can Perform Innumerable Contortions of Brain Data, and Then Claim a Discovery

Claims by neuroscientists that they have found "representations" in the brain (other than genetic representations) are examples of what very abundantly exists in biology: groundless achievement legends. There is no robust evidence for any such representations. 

Excluding the genetic information stored in DNA and its genes, there are simply no physical signs of learned information stored in a brain in any kind of organized format that resembles some kind of system of representation. If learned information were stored in a brain, it would tend to have an easily detected hallmark: the hallmark of token repetition.  There would be some system of tokens, each of which would represent something, perhaps a sound or a color pixel or a letter. There would be very many repetitions of different types of symbolic tokens.   Some examples of tokens are given below. Other examples of tokens include nucleotide base pairs (which in particular combinations of 3 base pairs represent particular amino acids), and also coins and bills (some particular combination of coins and bills can represent some particular amount of wealth). 

Other than the nucleotide base pair triple combinations that represent mere low-level chemical information such as amino acids, something found in neurons and many other types of cells outside of the brain, there is no sign at all of any repetition of symbolic tokens in the brain. Except for genetic information which is merely low-level chemical information, we can find none of the hallmarks of symbolic information (the repetition of symbolic tokens) inside the brain. No one has ever found anything that looks like traces or remnants of learned information by studying brain tissue. If you cut off some piece of brain tissue when someone dies, and place it under the most powerful electron microscope, you will never find any evidence that such tissue stored information learned during a lifetime, and you will never be able to figure out what a person learned from studying such tissue.  This is one reason why scientists and law enforcement officials never bother to preserve the brains of dead people in hopes of learning something about what such people experienced during their lives, or what they thought or believed, or what deeds they committed.    

But despite their complete failure to find any robust evidence of non-genetic representations in the brain, neuroscientists often make groundless boasts of having discovered representations. What is going on is pareidolia, people reporting seeing something that is not there, after wishfully analyzing large amounts of ambiguous and hazy data. It's like someone eagerly analyzing his toast every day for years, looking for something that looks like the face of Jesus, and eventually reporting he saw something that looked to him like the face of Jesus.  It's also like someone walking in many different forests, eagerly looking for face shapes on trees, and occasionally reporting a success, or like someone scanning the sky, looking for clouds that look like animal shapes.

The latest example of nonsensical neuroscientist pareidolia is to be found in a press release from Columbia University, and the paper that press release describes in very misleading terms.  The press release has the phony headline "Scientists Capture Clearest Glimpse of How Brain Cells Embody Thought." When you read a headline like that, you should remember a sad truth that has been glaringly obvious for many years now: the fact that university press releases on topics of science are no more trustworthy than corporate PR press releases.  There is the most gigantic amount of lying, hype and misrepresentation in university press releases these days, and such baloney occurs in equal amounts in the press releases of every major university. So don't think for a moment than you can  trust a press release because it came from Harvard or Columbia or Yale or Oxford University. I wish I had a dollar for every bogus press release that has been issued by such institutions. The subtitle of the press release is the 100% untrue claim "Recordings from thousands of neurons reveal how a person’s brain abstractly represents acts of reasoning."

We have an utterly groundless claim by a neuroscientist that he and his colleagues found a “uniquely revealing dataset that is letting us for the first time monitor how the brain’s cells represent a learning process critical for inferential reasoning." We then have an equally groundless claim by another neuroscientist that "this work elucidates a neural basis for conceptual knowledge, which is essential for reasoning, making inferences, planning and even regulating emotions.”

Do the authors claim to have seen some structure in the brain corresponding to these claims? Certainly not. They did not do any brain imaging such as using MRI scans. All they had to work with is EEG readings, readings of brain waves. Would anyone have seen any sign of such claimed representations by visually examining the wavy lines of these EEG readings? Certainly not. 

The press release reveals that what is going on is an affair that can be described as "keep torturing the data until it confesses in the weakest voice." We read this:

"The researchers recast the volunteers’ brain activity into geometric representations – into shapes, that is – albeit ones occupying thousands of dimensions instead of the familiar three dimensions that we routinely visualize. 'These are high-dimensional geometrical shapes that we cannot imagine or visualize on a computer monitor,' said Dr. Fusi. 'But we can use mathematical techniques to visualize much simplified renditions of them in 3D.' ” 

What a joke. They didn't see any such representations of knowledge or thought by a simple examination of the brain wave data they acquired. So they kept fiddling with their data and manipulating the data and contorting the data with some kind of absurdly convoluted and byzantine analysis pathway, and then claimed to see representations or shapes in such super-manipulated data. It's like someone taking 1000 pictures of the clouds in the sky, and then playing around all day with image manipulation filters until he got something that looked like an animal shape in one of the clouds. 

The paper being described (which you can read here) describes the huge chain of arbitrary manipulations and arbitrary naming that went on. It's a laughably arbitrary and byzantine analysis pathway in which many dozens of arbitrary analysis decisions are being made. 

Here from the paper is a description of just a small fraction of the "keep torturing the data until it confessed" spaghetti code craziness that went on:

" A cross-session-group PS was then computed by applying the same alignment to a pair of held-out conditions, one on either side of the current dichotomy boundary. Alignment and cross-group comparisons were performed in a space derived using dimensionality reduction (six dimensions). For a given dichotomy, two groups of sessions with N and M neurons were aligned by applying singular value decomposition to the firing-rate normalized condition averages of all but two of the eight task conditions, one on either side of the dichotomy boundary. The top six singular vectors corresponding to the non-zero singular values from each session group were then used as projection matrices to embed the condition averages from each session group in a six-dimensional space. Alignment between the two groups of sessions, in the six-dimensional space, was then performed by computing the average coding vector crossing the dichotomy boundary for each session group, with the vector difference between these two coding vectors defining the ‘transformation’ between the two embedding spaces. To compare whether coding directions generalize between the two groups of sessions, we then used the data from the two remaining held-out conditions (in both session groups). We first projected these data points into the same six-dimensional embedding spaces and computed the coding vectors between the two in each embedding space. We then applied the transformation vector to the coding vector in the first embedding space, thereby transforming it into the coordinate system of the second session groups. Within the second session group embedding space, we then computed the cosine similarity between the transformed coding vector from the first session group and the coding vector from the second session group to examine whether the two were parallel (if so, the coding vectors generalize). We repeated this procedure for each of the other three pairs of conditions being the held-out pair, thereby estimating the vector transformation of each pair of conditions independently. The average cosine similarity was then computed over the held-out pairs. All possible configurations of conditions aligned on either side of the dichotomy boundary are considered (24 in this case), and the maximum cosine similarity over configurations is returned as the PS for that dichotomy (plotted as ‘cross-half’ in Extended Data Fig. 3z)."

This is only a very small fraction of the "manipulate the data like crazy" nonsense that was going on. The paper lists a total amount of statistical rigmarole that seems ten times more complicated than what is described in the quote above. Never be impressed when you read about such operations. The more complicated such paragraphs are, the more it shows that the original data did not have the final result claimed, and that the authors had to play "keep torturing the data until it confesses" games, using a long series of arbitrary data manipulations  and data contortions to try to "gin up some success."  Strangely, we read almost nothing in the way of justifying these bizarre data manipulations and data contortions. It is as if the authors thought they had the right to dream up the most enormously  convoluted data manipulation scheme, without justifying the bizarre data distortions they were doing. 

A look at some of the programming code used shows that all the data was being passed through doubly-nested loops that were doing God-only-knows-what:

%for every area in the dataset
for i = 1:length(cell_groups)
    
    %run analysis for the inference absent sessions
    idx_current = intersect(cell_groups{i},find(ismember(sessions,inference_absent)));

    for i_rs = 1:n_resample
        [avg_array,~] = construct_regressors(neu,n_samples(i),idx_current);
    
        [t_1,t_2]        = sd(avg_array,n_perm_inner,n_samples(i));
        sd_{i,1}         = cat(2,sd_{i,1},t_1);
        sd_boot{i,1}     = cat(2,sd_boot{i,1},t_2);
        [ccgp_{i,1}]     = cat(2,ccgp_{i,1},ccgp(avg_array,...
                           n_perm_inner,false,n_samples(i)));
        [ccgp_boot{i,1}] = cat(2,ccgp_boot{i,1},ccgp(avg_array,...
                           n_perm_inner,true,n_samples(i)));
        ps_{i,1}         = cat(2,ps_{i,1},ps(avg_array,...
                           n_perm_inner,false));
        ps_boot{i,1}     = cat(2,ps_boot{i,1},ps(avg_array,...
                           n_perm_inner,true));
    end
    
    %run again for the inference present sessions
    idx_current = intersect(cell_groups{i},find(ismember(sessions,inference_present)));

    for i_rs = 1:n_resample
        [avg_array,~] = construct_regressors(neu,n_samples(i),idx_current);
    
        [t_1,t_2]        = sd(avg_array,n_perm_inner,n_samples(i));
        sd_{i,2}         = cat(2,sd_{i,2},t_1);
        sd_boot{i,2}     = cat(2,sd_boot{i,2},t_2);
        [ccgp_{i,2}]     = cat(2,ccgp_{i,2},ccgp(avg_array,...
                           n_perm_inner,false,n_samples(i)));
        [ccgp_boot{i,2}] = cat(2,ccgp_boot{i,2},ccgp(avg_array,...
                           n_perm_inner,true,n_samples(i)));
        ps_{i,2}         = cat(2,ps_{i,2},ps(avg_array,...
                           n_perm_inner,false));
        ps_boot{i,2}     = cat(2,ps_boot{i,2},ps(avg_array,...
                           n_perm_inner,true));
    end

end

Every single piece of data is being passed into a function called construct_regressors(), but what is that function doing? We cannot tell, because the code for that function has not been supplied.  We should be suspicious that this construct_regressors() function was doing something so arbitrary and convoluted that the authors were embarrassed to publish the code for that function. 

What we have here is something like the situation described in the visual below:

What we have here is something like the situation described in the visual below:

To pass off the results of so vast an amount of data monkeying as a discovery is a case of BS and baloney. No representations in the brain or brain waves have been discovered here. All we have is scientists manipulating and contorting data like crazy, and then displaying some pareidolia by passing off their super-manipulated data as an example of "representations."  

Can you imagine what a scandal would arise if climate scientists tried to get away with even one tenth of this amount of manipulating and contorting and distorting their data? Skeptics of their work would start "screaming bloody murder," and howl about how scientists were failing to use their original data, and using instead manipulated, contorted, twisted, distorted data.  But it seems that neuroscientists senselessly think that it is okay for them to play around endlessly with data from brains,  and that they have the right to contort and twist and distort such data in dozens of different ways, and then pass off the result (an utterly artificial construction) as something they can then call "what the brain does."  

What can you call data like this which has undergone so many contortions and manipulations and distortions and transfigurations that it is something almost totally different from the raw data originally gathered? You might be tempted to call it "fake data," but that isn't quite right, because the authors have described all the transformations they did of the data. So rather than calling it "fake data," a description that is not quite right, we can merely say that it is data that has been so enormously contorted and manipulated and transfigured that it is data that cannot be claimed as evidence telling us about brain states or brain activity. 

Experimental neuroscience is in a state of great sickness and dysfunction. The use of Questionable Research Practices seems like more the rule in experimental neuroscience than the exception. When scientists think that it is okay for them to perform endless manipulations and contortions and distortions and transfigurations of their data, and to then pass off the resulting artificial mess as "what we got from the brain," then it is a sign that neuroscience has sunk to an extremely low nadir of dysfunction. 

Brain waves don't represent anything that someone has learned. Brain waves are no more representations of something learned than clouds are representations of something learned. Brain waves are streams of random data, as random as the stream of clouds passing above a house or  a city.  Not one iota of evidence of brain representations has been presented in this paper. The paper is entitled "Abstract representations emerge in human hippocampal neurons during inference." An honest title for the paper would have been "We got something we called 'abstract representations'  after we manipulated and contorted brain wave readings in dozens of weird ways."

For additional examples of neuroscientists using computer programs to play "keep torturing the data until it confesses," read my post here, entitled "Programming Gone Astray: Iteration Inanity of the Neuroscientists' Distortion Loops."

Contortion craziness