Neuroscientists Keep Writing Papers Bluffing Us About Memory Encoding

 " We do not yet have a systematic theory of how information is encoded in the chemical and electrical activity of neurons, how it is fused to determine behavior on short time scales, and how it is used to adapt, refine, and learn behaviors on longer time scales. " -- "BRAIN 2025: A Scientific Vision," document co-authored by many scientists (link). 

If we are to believe in the claim that brains store human memories, we must have a credible account of four things: encoding, neural storage of very old memories, the instantaneous formation of memories, and the instantaneous retrieval of memories. The theory that human memories are stored in the brain fails in regard to each of these things.

There exists no plausible theory as to how a brain could store memories lasting for 50 years, but we know humans can remember many things for that long. The most popular idea of brain memory storage claims that memories are stored in synapses, but the proteins in synapses have an average lifetime of less than two weeks, meaning such a theory falls short by a factor of 1000 when it comes to explaining memories that persist for 50 years. As for memory retrieval, there is no theory explaining how humans could possibly recall instantly things they learned many years ago, and haven't thought about in years. 

You may hear the name of some obscure historical or cultural figure you learned about decades ago, and haven't heard about or thought about since that time. You may then instantly recall something about that person. But if that memory was stored somewhere in your brain, how could you instantly find the exact little location where that memory was? Doing that (for example, instantly finding a memory in storage spot 834,220 out of 1,200,000) would be like instantly finding a needle in a mountain-sized haystack. If a brain had an indexing system, or a coordinate system, or a neuron numbering system, there might be a faint hope for explaining instantaneous memory retrieval; but the brain has no such things. As for the instantaneous formation of memories, there is no theory that can account for it in a brain. The prevailing theory that memories are stored by synapse strengthening (which would involve protein synthesis requiring minutes) fails to account for memories that humans can form instantly.

When we consider the issue of memory encoding, we find a difficulty as great as the difficulties just discussed. Encoding is supposedly some translation that occurs so that a memory can be physically stored in a brain, so that it might last for years. The problem is that human memories include incredibly diverse types of things, and we have no idea how most of these things could be stored as neural states. Consider only a few of the types of things that can be stored in a human memory:

• Memories of daily experiences, such as what you were doing on some day
• Facts you learned in school, such as the fact that Lincoln was shot at Ford's Theater
• Sequences of numbers such as your social security number
• Sequences of words, such as the dialog an actor has to recite in a play
• Sequences of musical notes, such as the notes an opera singer has to sing
• Abstract concepts that you have learned
• Memories of particular non-visual sensations such as sounds, food tastes, smells, pain, and physical pleasure
• Memories of how to do physical things, such as how to ride a bicycle
• Memories of how you felt at emotional moments of your life
• Rules and principles, such as “look both ways before crossing the street”
• Memories of visual information, such as what a particular person's face looks like

How could all of these very different types of information ever be translated into neural states so that a brain could store them?

Our neuroscientists have told us again and again that the brain does such an encoding, but there is no real evidence that any such thing takes place. What we have evidence for is merely evidence that humans remember things. If you are someone who believes that memories are physically stored in brains, then you may claim that memory encoding occurred at such and such a rate whenever you observe people learning something at such and such a rate. But merely observing evidence of learning or memory is not acquiring any actual evidence that encoding has occurred. There remains the possibility that our memories are not stored as neural states, the possibility that our repository of memory is some spiritual or psychic facility that is non-neural and non-biological.

Such a possibility should not seem remote when we consider that there is no workable theory as to how learned knowledge and experiences could be encoded so that they might be stored in a brain. No matter what theory we may create to account for the encoding of learned knowledge and episodic experience so that they can be stored in a brain, such a theory will always end up sounding ridiculous after we examine the theory in detail and consider its requirements and shortcomings. Let's look at some possibilities, and why they fail.

Theory #1: Direct writing of words and images

First, let's consider the simplest theory of encoding we can imagine – that a memory is stored in the brain so that it appears in a neural form pretty much as we see it in our minds. Under this theory, when you memorized some series of words, this would cause a sequence of microscopic little letters to become stored in your brain; and when you experienced some visual experience, this would get stored as some tiny little image in your brain. So, for example, under this theory, if someone memorized the sentence, “There may be green aliens in the center of the galaxy,” then after the person died, some scientist might examine that person's brain with an electron microscope, and actually find some tiny little words in some neurons, words that directly spelled out, “There may be green aliens in the center of the galaxy.” And under this theory, if someone was given a picture of a toy purple pony, and asked to memorize it, then after the person died, a scientist might be able to examine the person's brain under an electron microscope, and the scientist might say, “Aha, I see in his neurons a tiny little image of a toy purple pony.”

This theory may immediately provoke giggles, and it is rather easy to think of some reasons why it does not work. They are these:

1. If memory worked in such a way, we would surely have already discovered such easily-recognizable memory traces. But no such things have been seen, even though a great deal of human neural tissue has been examined at very high magnification. When we look at brain tissue at the highest magnification, we see no tiny little letters or tiny little images of animals, cars, and persons.
2. For a brain to be able to write words that we memorized in this type of direct manner, it would seem that the brain would need some very precise write mechanism, capable of forming the exact characters of the alphabet in brain tissue; but no such brain capability is known to exist.
3. It seems that if such a theory were true, recalling some words would be like reading. But recalling words is almost never like reading, and we don't see in our mind's eye some stream of letters as we recall some words we memorized.
4. For a brain to be able to read words that we memorized in this type of direct manner, it would seem that the brain would need some very precise reading mechanism, capable of reading the exact characters of the alphabet stored in very tiny letters written in brain tissue; but no such thing is known to exist. We don't have tiny little “micro-eyes” in our brains that might allow us to read tiny microscopic letters stored in our brains.
5. Alphabets of modern languages are relatively recent inventions. For example, the English alphabet is not many  thousands of years old. There has no been no change in brain structure for many thousands of years, and we can think of no  reason why brains would have some ability allowing it to write an alphabet that only came into existence centuries or a few thousand years ago, not many thousands of years ago. 

Theory #2: Brain storage of words and images using some unknown non-binary coding or translation protocol

Now, let's consider a different theory of memory encoding – the idea that instead of directly storing words and images (so that we could directly read the words and directly see the images), the brain uses some type of unknown coding or translation protocols. For example, it could conceivably be that words that we learn are somehow translated into proteins or chemicals or electrical states, using some as-of-yet undiscovered translation scheme.

For example, such a scheme might work a little like this:

Item	How the item might be represented
Letter “A”	Some particular neural arrangement of atoms, chemicals or electricity
Letter “B”	Some other neural arrangement of atoms, chemicals or electricity
Letter “C”	Some other neural  arrangement of atoms, chemicals or electricity

Such a scheme might work a little like the Morse code, in which particular letters are translated into some sequence of dots, dashes, or dots and dashes. Some particular arrangement of atoms, chemicals or electricity might work like a dot in the Morse code, and some other particular arrangement of atoms, chemicals or electricity might work like a dash in the Morse code.

Or there could be some higher-level translation system based on particular words rather than letters. For example, we can imagine something like this:

Item	How the item might be represented
Word “sun”	Some particular neural arrangement of atoms, chemicals, proteins or electricity
Word “man”	Some other neural arrangement of atoms, chemicals, proteins or electricity
Word “move”	Some other neural  arrangement of atoms, proteins chemicals or electricity

There is one giant problem with such a theory. All of the languages that we use are fairly recent innovations, having been created in only the last few percent of the time that humans have existed. For example, back in the Roman Empire people used Latin, but the English we use today has only been in use for less than 1200 years. The alphabet used for English is less than 1000 years old, and its alphabetic predecessor (the Latin alphabet) is only a few thousand years old. It is generally acknowledged even by Darwinism enthusiasts that very complex evolutionary innovations cannot arise in only a few centuries of time or a few thousand years. So we could never explain how the brain could naturally possess some elaborate translation system based on such a relatively recent innovation as the English language and the English alphabet.

Scientists strain our credulity whenever they talk about novel functional genes accidentally appearing even over the course of a million years. Think, then, on how much greater a problem there would be in explaining how hundreds of novel functional genes could have appeared in less than 3000 years, to perform some translation operation involving characters and words that have existed for less than 3000 years. To assume such a thing would be to assume evolution working thousands of times faster than the rate we would predict from known mutation rates.

There is also no evidence that any such great burst of genetic novelty has occurred. Although the half-life of DNA is 512 years, we have enough samples of human DNA from ancient Rome and ancient Egypt to know that there has been no big change in the DNA of humans during the past 3000 years. So it seems impossible that there could be any genetic capability (arising in the past few thousand years) that would allow humans to neurally store information using some encoding mechanism specifically tailored to the letters and words of the English language that have existed for less than 3000 years.

Another difficulty with the theory of encoding just mentioned is that if it existed, we would see big differences in the genes of people who spoke different languages. According to such a theory, we would expect that Chinese people would have one group of genes corresponding to proteins or RNA molecules needed to translate Chinese words into neural states, and that English speaking people would have some other quite different set of genes corresponding to proteins or RNA molecules needed to translate English words into neural states (particularly since the Chinese language and alphabet is so different from the English language and alphabet). But there exists no such difference in the genes of Chinese speaking people and English speaking people.

There is also the difficulty that there is no sign in the human genome that any such genes exist for performing such an elaborate operation of encoding human learned knowledge and episodic experience so that it can be stored in neurons or synapses (and there would need to be many hundreds or thousands of types of genes dedicated to performing such a task if it was done).

Theory #3: Binary writing of words and images

Now, let's consider a theory of memory encoding that perhaps the words we memorize and the images we remember are stored in binary format. We know that computers store information in binary format, so when it is suggested that the brain may use a similar format, this may sound reasonable to the average person (although it isn't, a brain being radically different from an electronic computer).

This possibility actually has all of the difficulties of the previous possibility. What goes on when your computer stores words in binary format is the following:

1. First individual letters in the words are converted into decimal numbers (such as 13, 19, and 23) using a particular translation table called the ASCII code.
2. Then, those numbers are converted from decimal to binary using a decimal-to-binary conversion routine.

So if we are to believe that the brain does binary encoding like a computer, we would need to believe that built into the brain on a low level is some type of translation scheme like the one below, a scheme in which letters are translated into decimal numbers.

ASCII table used by a computer to store encoded information

In addition, we would also have to believe that the brain has some kind of capability to translate the numbers in such a system into binary numbers. Alternately, we could believe that the brain has a scheme for directly translating characters into binary, but the overall complexity of such a translation mechanism would be every bit as great as a system in which characters are converted into decimal, and then into binary.

We have the following difficulties involved with such an idea:

1. If memory worked in such a way, we would surely have already discovered such easily-recognizable memory traces. We would have discovered tiny little traces in the brain that resemble binary coding. But no such things have been seen, even though a great deal of neural tissue has been examined at very high magnification.
2. For a brain to be able to write words that were memorized in this type of direct manner, it would seem that the brain would need some very precise write mechanism, capable of writing binary traces; but no such thing is known to exist.
3. For a brain to be able to read words that we memorized in this type of direct manner, it would seem that the brain would need some very precise reading mechanism, capable of reading in binary; but no such thing is known to exist.
4. Since the alphabets of human languages are only a few thousand years old, there would have been no time for the human body to have evolved some complex biological mechanism capable of converting specific alphabetic characters to binary.
5. We can imagine no way in which a brain could achieve the translation effect in which words are translated into binary. As far as we know, there is nothing anything like an ASCII table in your brain, nor is there anything like a facility for translating English letters directly into binary, nor is there anything like a facility for translating English letters into decimal, and then translating decimal numbers into binary. There are no genes in the genome that perform such tasks.

The Scientists Who Keep Falsely Talking About Memory Encoding and Memory Representations in Brains

The simple fact is that there is not the slightest evidence of memory encoding or memory representation in the human brain. So why it that we keep getting scientific papers claiming to have seen memory encoding or memory representation in the human brain? It's pretty much a "they're talking like they had what they need to have but don't have" kind of affair.  Something similar might be going on if some street beggar were to try to attract women by frequently using finance terms. So someone without $20 to his name might speak like this:  "I think that WEALTH MANAGEMENT is very important, and I am slowly BUILDING MY FORTUNE in a way that will give me a HUGE BONANZA OF WEALTH once my INVESTMENT PLANS are actualized."

But how can someone write a paper making frequent references to memory encoding and memory representation in the brain, even though there is no evidence that such things exist?  The visual below shows the general approach typically used. 

A key element is what can be called chance-mining. Chance-mining typically involves grandly trumpeting some little result that might easily be obtained by pure chance.  I can give an example of chance-mining. Suppose you are doing a study trying to prove what we may call "the Abracadabra Hypothesis," which you define as "the hypothesis that saying 'Abracadabra' before flipping a coin will increase the chance of a Heads result." Any test involving a sufficiently large sample size will fail to support this Abracadabra Hypothesis. That's because something called the Law of Large Numbers says that the more random trials you do, the more the result will tend to match the result expected by chance. But it is very easy to get a test result deviating from what is expected by chance, if you use a very small sample.  For example, it is easy to flip ten coins and get 6 or 7 Heads results, even though the expected chance result is 5.

So if you want to do a paper claiming to support the Abracadabra Hypothesis, you can try a test using a small number of coin flips such as only 10 or 20. If that fails, you can file the results in your file drawer, and start a new experiment, also using a small number of coin flips. Probably before long you will have some result you can claim as "statistically significant." Chance-mining like this misleads people, because it creates the impression that there is a causal relation or causal influence when there is none. Saying "Abracadabra" actually has no influence on the result of coin flips. 

Similar chance-mining occurs in quite a few neuroscience papers.  The papers may make quite a few references to neural encoding or neural representation, but they will not be backed up by references to well-designed studies using an adequate sample size.  Typically the paper will present some research that is utterly unreliable for multiple reasons, such as the use of way-too-small study group sizes or unreliable techniques for trying to measure recall in rodents. 

A example of such a paper is the paper here, entitled "Ventral hippocampus neurons encode meal-related memory." The title is unjustified. No robust evidence has been provided of any neural encoding of memory. There are two gigantic flaws in the study:

(1) The study group size is way too small, consisting of only 5 animals. 

(2) No reliable method was used to measure memory performance in rodents. We read that a "foraging-related spatial memory task" apparatus was constructed to test memory in mice. We see a picture of a weird device that the paper does not name. This was not any well-established protocol for testing memory in mice. We fail to have any diagram explaining why such a device would reliably test memory in mice. 

What usually occurs in such papers claiming evidence of "memory encoding" or "memory representation" is that there will be a combination of two main sins, either one of which is to disqualify a paper as good evidence:

(1) Almost always there will be some study group size that is way too-small for any reliable evidence to be honestly claimed 

(2) Usually there will be a reliance on a completely unreliable measurement for estimating fear or recall in rodents, the method of trying to judge "freezing behavior." All papers relying on this method are junk science, for reasons I discuss here. 

What largely goes on in such poorly written science papers is pareidolia.  Pareidolia is what occurs when someone keeps scanning some stream of random data, looking for something he can call a representation. An example is a person who eagerly scans his toast every day, looking for some sign of a representation, and who one day announces he has found the face of Jesus in his toast, or maybe the face of the Virgin Mary, or maybe the face of his dead wife. 

The neuroscientist eagerly scanning brain scan data or brain wave data looking for something he can call a little bit of "representation" or "encoding" is someone who we can no more trust than someone eagerly scanning the clouds every day, looking for something that looks like the shape of an animal. 

An example from 2025 of a pure pareidolia paper is the junk science paper "Movie-watching evokes ripple-like activity within events and at event boundaries." We have scientists looking at EEG readings obtained in ten epilepsy patients while the patients watched movies.  Looking at the ever-changing line squiggles that are EEG readings, the authors claim to have seen "ripples" that they claim occur more often during "event boundaries." The claim that there are such "ripples" in the data is as much see-what-you-want-to-see pareidolia as the claim that there are particular points in movies that are "event boundaries." The supposed identification of these "ripples" occurred not by humans staring at the EEG readings, but by some computer program which is a black box of convoluted spaghetti code. Computer programming is the best friend of the scientist trying to see things that are not really there. You can run some computer program that is supposedly seeking some obscure thing in noisy data; and if the first analysis run fails, you can keep tweaking the code until it reports finding what you were hoping to find. 

How to Tell Whether a Neuroscience Research Announcement Is Unjustified

Questionable Research Practices and shoddy methods are extremely abundant in today's neuroscience research.  How is it that you can detect such examples of poor research? I will give here a method. The method mainly involves looking for certain types of defects in scientific papers, but also involves looking for defects in press announcements about such papers. 

Step #1: open up a "defect list" file

You will be using this file to record any defects you find in either the original scientific paper or any of the press announcements that occur about that paper.  You can create such a file by getting a blank piece of paper, opening up a new file using a tool such as Notepad, opening up a file using Google Docs, clicking on the Notes utility on your I-Pad, and so forth. 

Step #2: Look for a claim in the headline of an article or press release announcing the research that is not justified by any claims in the text of the article or the text of the academic press release. 

An article that you read announcing the research may or may not be the academic press announcing the research. If the press article is not the original academic press release, it may have a link to the academic press release.  The academic press release will typically have a link to a newly published scientific paper, and such a link may also be found in some article based on the press release. 

An extremely common defect of press articles about scientific papers is that they very often make boastful headline claims that are not justified by anything claimed or established in the articles underneath such headlines. This often occurs for economic reasons, to create the effect known as clickbait. Clickbait is when online articles have sensational-sounding headlines or interesting-sounding headlines that lure you into going to some web page that has ads. The people running or funding such pages thereby get advertising revenue when such pages are viewed. 

Clickbait is enormously abundant in the world of science journalism. University press releases very often contain headlines never justified by anything mentioned in the story underneath such headlines. Press articles based on such press releases very often contain headlines never justified by anything mentioned in the story underneath such headlines, or never justified by anything stated in the body of the press release such articles were based on.

A simple starting point in detecting unjustified neuroscience research announcements is to simply compare headlines to the text underlying such headlines, and note cases in which the headline is unjustified. Record any such cases in your "Defects list" file, nothing the URL of the corresponding press article or press release.

I will give a very simple example of such a thing. A headline may announce "Scientists Unlock the Secret of Human Memory Retrieval" But the underlying story may refer to research that only dealt with mice. In such a case the unjustified hype is obvious -- the research told us nothing about human memory. 

Step #3: find a copy of the scientific paper that is the basis of the press release or press article. 

Generally the press article or press release will have a link to a scientific paper that is the basis of the research announcement. In the easiest case, you will simply be able to click on that link to get the full text of the paper.  But in many cases when you click on the link, you will go to a page that merely has the abstract of the paper. There may be some "Full Text" link that asks you to pay money. Such a barrier to you reading the paper is called a paywall. I strongly advise against ever paying money merely to research the quality of a neuroscience paper.  Most neuroscience research papers these days are poor quality papers guilty of multiple examples of Questionable Research Practices. 

But if you find yourself blocked by a paywall, there are some things you can do to try to get the paper:

(1) Go to the Google Scholar site (https://scholar.google.com/), and copy the name of the paper into the search bar. See whether the paper shows in the search results, with a link to the full text of the paper.

(2) Go to the biology preprint server (https://www.biorxiv.org/) and copy the name of the paper into the search bar. See whether the paper shows in the search results. If it does, you will typically be able to get the full text of the paper by clicking on the Full Text tab on that site. 

Step #4: examine the title of the paper and read its abstract, looking for a claim in the title that is not matched by any claim in the abstract

It is surprisingly common these days for the titles of neuroscience research papers to make claims that are not justified by any statements made in either the abstract of the paper or the full text of the paper. If you find any discrepancy between the title of the paper and the results announced in the abstract, record such a discrepancy in your "defects list" file.

Step #5: if the paper is an experimental research file, look for evidence of insufficient study group size

Since the use of way-too-small study group sizes is amazingly predominant these days in experimental neuroscience research,  the abstract of every paper should tell how many subjects were used in each of the study groups. But like people who are trying to hide their shortcomings, the abstracts of experimental neuroscience research papers rarely list the study group sizes used. So you will usually need to search the text of the paper for an indication of the study group sizes used.  

Any experimental neuroscience paper using fewer than 15 subjects in any of its study groups should be regarded as a paper that has used a way-too-small study group size. There are actually reasons for thinking that any use of fewer than 25 subjects in any of the study groups is a reason for doubting the quality of the study, particularly if the work involves brain scans of humans.  

Do not stop looking for study group sizes if you see a statement indicating a fairly large study group size such as 50. What very often happens in neuroscience papers is that the paper will announce a fairly large number of subjects (such as stating "50 mice were analyzed"), but will then divide this group up into smaller study groups, so that the smallest study groups used is much smaller than such a fairly large number. Look for any cases of any study group sizes smaller than 15. 

How do you find what study group sizes were used? The easiest way is to search in the text for the phrases "n=" or "n =". It is a custom in neuroscience research papers to list study group sizes using phrases such as "n =8." For example, the text may vaguely refer to "subjects" or "mice" without specifying how many. Then the text of the paper may state the exact number of subjects by using a phrase such as "n = 10." Another way to search for study group sizes is to search for the phrases "mice," "rats," "subjects" or "humans" and look for a number preceding such phrases. 

Whenever any such searches reveal a study group size of less than 15 or 20, you have discovered prima facie evidence of a too-small study group size, and you should record such a defect in your "defects list" file. Rarely you will find a paper that makes no mention of how many experimental subjects were used. The failure to record so vital a fact is itself a defect that you should note in your "defects list" file.

When an experimental neuroscientist is doing his job right, he will use some statistical method to do what is called a power analysis or a sample size calculation or a power size calculation. This involves some mathematical calculation of what sample size was needed to achieve some particular degree of statistical power.  Many science journals require that a paper state whether or not such a calculation was done. Search for the phrase "sample size calculation" or "power size calculation" or "power calculation" to see whether such a calculation was done. You will often read a confession that no such calculation was done.  If you find such a confession, write that down in your "defects list" file.  If you fail to find any mention of such a calculation, that is also a defect that should be noted in your defects list file. 

Step #6: look for a failure to use controls

Almost any experimental neuroscience experiment should be using controls.  In experimental science a control can be a subject that does not have some characteristic or variable or intervention being tested, or a control can be a neutral state that does not match some experience or condition being tested.  For example, if you are testing some medicine, you can give 15 subjects the medicine, and give 15 other subjects some placebo that is not some medicine. Or, if you are testing whether some cognitive activity such as memory recall causes increased activation of some brain region, you might take 15 brain scans while a subject was engaging in memory recall, and 15 brains "control" scans on some other day in which the subject was asked to think of nothing.

It is easy to check whether a scientific study made use of controls. Just do a text search in the paper for the word "control" looking for usage that indicates controls were used. Typically phrases such as "control subjects" or "control state" will be used.  If you fail to find any evidence controls were used, record that failure in your "defects list" file.

Step #7: look for a failure to follow a detailed blinding protocol

In experimental neuroscience a blinding protocol is usually needed for a robust result. A blinding protocol is a procedure that helps to minimize the chance of a biased analysis.  I can give some examples to illustrate the concept. Imagine you brain scan 15 subjects who were asked to recall memories while their brains were scanned, and you also brain scan 15 other subjects who were asked to think of nothing while their brains were being scanned.  Then suppose you give the brain scans to some analyst.  If the analyst knows that the first 15 brain scans were from people who were recalling things, and the second 15 subjects were from people thinking of nothing, and also that the purpose of the test is to look for brain differences in memory recall, such an analyst will be all too likely to "see what his bosses are hoping he sees," and produce a biased result.  The risk of such bias could be avoided by a careful blinding protocol.  Each set of brain scans could be assigned a random number, with someone recording which number matched a person recalling something, and which number corresponded to a person not recalling something.  If there was a stack of 30 folders, each containing one subject's brain scans, the folders could be shuffled so that the analyst could not tell which one came from some one recalling something. It could then be stated in the paper that the person analyzing the brains was "blind to which subjects had engaged in memory recall." 

There is another type of blinding that could occur. Instead of the brain scan analyst being told that half of the subjects were engaging in memory recall, and half were not, the analyst could be told nothing at all about what the people were doing during the scans. This would make it all the more unlikely that the analyst would see some effect that wasn't really there.  Normally in every study there are multiple ways in which blinding should occur. 

A failure to follow blinding protocols is one of the most egregious defects of today's neuroscience research. Most experimental neuroscience studies fail to follow any blinding protocol. The failure can easily be found if you have the full text of the paper. Simply do a text search for the word "blind." If you fail to find meaningful uses of the word "blind" in the text of the paper, thereby indicating a failure to follow a blinding protocol, record that failure in your "defects list" file.

One or two uses of the word "blind" in the text of a paper does not show that an effective blinding protocol was used. It is all too easy for some experimenter to have an ineffective blinding protocol that fails to achieve much of any real blinding. The smaller the study group size, the easier it is for any attempt at blinding to be ineffective.  I will give an example. Suppose a study used only seven rats for an experimental group, and seven rats for a control group.  The seven rats in the experimental group might be given some modification not given to the seven rats in the control group.  After being given foot tags to identify them with random numbers, the fourteen rats might then be given to an analyst asked to test for some difference.  But if the analyst was involved in applying the modification, it might be all too easy for him to recognize which rat had the modification, and which did not.  For example, only the rats given the modification might have some surgical mark showing they had the modification. What this example shows is that to be effective, a blinding protocol must be a detailed, carefully thought-out plan that prevents "sham blinding" in which someone supposedly blind to which subjects were in the control group is not really blind to such a thing.  

If you either fail to see the word "blind" being used in the text of an experimental neuroscience paper, or if you find the word "blind" or "blinding" only being used once or twice "in passing," you should note the result in your "defects list" file. Also note it in your "defects list" file if you fail to find a detailed discussion of a blinding protocol. If a study merely claims that some analysis was done by analysts "blind" to whether the subjects were control subjects, and fails to discuss how a careful detailed plan was followed to prevent "sham" blinding, that also should be recorded in your "defects list" file. 

Step #8: look for convoluted analysis pathways that may have "conjured up" some illusory result

In my post "Convoluted 'Spaghetti Code' Analysis Pathways Help Neuroscientists Conjure Phantasms That Don't Exist," I gave two examples of scientific papers that used ridiculously convoluted analysis pathways.  Don't be impressed when you see such methods, which may seem like gobbledygook or rigmarole.  Such over-complicated methods are typically not signs of good experimental methods, but instead a failure to follow a straightforward technique for analyzing data. What is going on often can be describe as "keep torturing the data until it confesses." 

Besides doing a quick scan looking for byzantine analysis pathways that sound like statistical "monkey business," there are some things you can look for:

(1) Look for the word "iterations" which typically indicates that some data was passed through a programming loop in which the data may have been distorted or contorted or convoluted. 

(2) Look for the phrases "processing" (particularly "data processing," "preprocessing" or "pre-processing" or "post-processing" or "postprocessing"), phrases which indicate that data has been passed through some computer program.  Once data has been passed through a computer program, there are any number of ways in which the data can be contorted, distorted or corrupted.  Often computer programs processing neuroscience data are written by scientists who are not professional computer programmers, and who can often produce unreliable results. If the programming job is given to a professional programmer, the person may be someone who does not understand the data, leading to unreliable results. Programming code used in scientific research is very often poorly written and poorly commented, producing effects that may be known only to the original programming. Very often the result is a "black box" situation in which even the original programmer does not understand what is happening to the data. 

If you find evidence of such dubious-sounding analysis pathways or dubious pre-processing or post-processing of brain scan data or neuroscience data, add some lines to your "defects list" file recording such a finding. 

Step #9: look for fake data in the science paper, which may be described using the word "simulated" or "simulation." 

Neuroscientists often look for things they cannot find but are hoping to find. When they cannot find such things, they often resort to generating simulated data to try to fill in the gap. Whenever you hear the word "simulated" in a neuroscience paper, you should presume that this actually means "fake."  Search for the words "simulated" or "simulation" in a neuroscience paper. Note the resorting to such simulated data in your "defects list" file. Trying to pass off simulated data rather as if it was real-world data is a sleazy trick you should "throw a flag on" when critically analyzing neuroscience papers. 

Step #10: look for p-hacking 

Modern experimental science has the silly rule that a result is treated of worthy of publication if someone found it to have a "statistical significance" of .05 or less.  This rule ends up being a very silly one in many cases in which it is very easy to get such a result when it is merely a false alarm. Very roughly you can think of a statistical significance of .05 as a result you might get by chance once in 20 tries. Given a lack of pre-registration in scientific studies, it is rather easy to get such a result. You can just keep trying something multiple times, calling these trials Experiment 1, Experiment 2, and so forth. When you get a result that you would get by chance once in 20 times, you can then write up that result, and describe only it. 

How do you find evidence of unimpressive results such as this in a scientific paper? Statistical significance is reported using a phrase such as "p < .05" or "p < .01" or "p < .001."  You can search for such phrases. Whenever you find the phrase "p < .05"  it is a sign that an unimpressive result was obtained.  Write down any such occurrence in your "defects list" file. The more examples you find of the phrase "p < .05" the stronger the case you can make that "p-hacking" went on. 

Don't be impressed if you find a stronger statistical significance reported in addition to a marginal statistical significance of "p < .05." What often happens is that the most relevant result will be some marginal borderline result of "p < .05" but other results not very relevant will be reported with a stronger statistical significance such as "p < .01" or "p < .001." This is often done as a kind of window dressing to give you the impression that the paper has more impressive results than it has. The results with higher statistical significance may be irrelevant to the main claim made by the paper. 

Step #11: look for lack of pre-registration 

It has been pointed out many times that when an experimenter fails to state before gathering data a hypothesis to be tested and a detailed research plan for how to gather and analyze data, there will be a much higher chance of some false alarm being reported. A researcher who is free to "make up his method as he goes along" will be free to keep playing around with data analysis pathways until he seems to find something he was hoping to find.  Pre-registration (also called the use of "registered reports") is when a researcher publishes a hypothesis to be tested and a detailed research plan before any data is gathered. It is widely recognized that following such a method greatly reduces the number of false alarms that are reported. 

It is easy to search in a scientific paper for whether pre-registration occurred. Simply search in the text of the paper for the terms "registered report," "pre-registered" or "pre-registration." If no such terms are found, note this in your "defects list" file. 

Step 12: look for unreliable measurements of memory, such as attempts to judge "freezing behavior." 

There are reliable ways to judge whether an animal remembered something, and also unreliable ways. A reliable way to judge whether a a rodent trained to fear some stimulus (such as a shock plate) is to measure heart spikes when the animal is placed near the pain-inducing stimulus.  Heart rates very dramatically spike in rodents when they are afraid. Another reliable way to judge whether a a rodent trained to fear some stimulus such as a shock plate is to use some setup such as the one shown below, in which an animal remembering the fear stimulus will take the harder path towards a reward rather the easier path.

A very unreliable way of measuring rodent recall of a fearful stimulus is to try to judge "freezing behavior," defined simply as immobility. My post here explains why such a technique is very unreliable as a way to judge whether recall or fear occurred.  Attempts to judge so-called "freezing behavior" are massively used in experimental neuroscience involving rodents and memory. This is an example of a dysfunctional research community tradition. If you find that the neuroscience paper you are examining made any use of judgments of "freezing behavior," note that in your "defects list" file.  All neuroscience papers relying on "freezing behavior" judgments are junk science.  

Once you have taken such steps, you have the basis for a critical review of a neuroscience paper.  Very often your "defects list" file will contain multiple examples of Questionable Research Practices, often more than five. Mentioning such defects, you can explain to a reader why some triumphal announcement of neuroscience progress is unjustified. 

Click to see left column more clearly

Click to see left column more clearly

In 2016 Scientists Confessed Science Research Was Broken, and in 2026 Things Are No Better

 A 2016 story article in Vox.com was a remarkable confession of how bad things are in the world of scientific research. The article was entitled "The 7 biggest problems facing science, according to 270 scientists."  We read this:

"We heard back from 270 scientists all over the world, including graduate students, senior professors, laboratory heads, and Fields Medalists. They told us that, in a variety of ways, their careers are being hijacked by perverse incentives. The result is bad science....Today, scientists’ success often isn’t measured by the quality of their questions or the rigor of their methods. It’s instead measured by how much grant money they win, the number of studies they publish, and how they spin their findings to appeal to the public."

The author writes about what is called publication bias, the tendency of science journals to prefer publishing studies that report finding some positive effect, rather than studies that fail to report such an effect, finding what is called a null result.  We read this: 

"Scientists often learn more from studies that fail. But failed studies can mean career death. So instead, they’re incentivized to generate positive results they can publish. And the phrase 'publish or perish'  hangs over nearly every decision. It’s a nagging whisper, like a Jedi’s path to the dark side."

The statement won't be very clear to the average reader. What the author means by "studies that fail" is not studies that fail to follow good science practice or that fail to be completed, but instead studies that fail to report a positive result. An example would be a study testing for whether removing one particular gene from mice has an effect on their memory, and which reports no effect from such a removal. By "failed studies can mean career death" the author means studies that failed to report a positive effect and that then did not get published, because of publication bias in which positive results are preferred.  A scientist doing enough of such studies that were not published might end up with a low count of published papers. 

We read about conflicts of interest:

"Already, much of nutrition science, for instance, is funded by the food industry — an inherent conflict of interest. And the vast majority of drug clinical trials are funded by drugmakers. Studies have found that private industry–funded research tends to yield conclusions that are more favorable to the sponsors."

Such conflicts of interest taint neuroscience research, because a large fraction of neuroscience research is funded (directly or indirectly) by pharmaceutical companies and biotech device manufacturers hoping to produce some result they can claim is scientific evidence in favor of some pill or device they are selling.  We read in the article that some professors are spending up to 50% of their time writing research grant proposals. 

We get this quote:

"'As it stands, too much of the research funding is going to too few of the researchers,' writes Gordon Pennycook, a PhD candidate in cognitive psychology at the University of Waterloo. 'This creates a culture that rewards fast, sexy (and probably wrong) results.' ”

A culture that rewards wrong results? How messed up is that?

We read the following:

"The problem here is that truly groundbreaking findings simply don’t occur very often, which means scientists face pressure to game their studies so they turn out to be a little more 'revolutionary.'  (Caveat: Many of the respondents who focused on this particular issue hailed from the biomedical and social sciences.)

Some of this bias can creep into decisions that are made early on: choosing whether or not to randomize participants, including a control group for comparison, or controlling for certain confounding factors but not others....Many of our survey respondents noted that perverse incentives can also push scientists to cut corners in how they analyze their data.

'I have incredible amounts of stress that maybe once I finish analyzing the data, it will not look significant enough for me to defend,' writes Jess Kautz, a PhD student at the University of Arizona. 'And if I get back mediocre results, there’s going to be incredible pressure to present it as a good result so they can get me out the door. At this moment, with all this in my mind, it is making me wonder whether I could give an intellectually honest assessment of my own work '.”

We read a quote from a Joseph Hilgard who says, "The scientist is in charge of evaluating the hypothesis, but the scientist also desperately wants the hypothesis to be true.”  We read the claim that 85 percent of research is "routinely wasted on poorly designed and redundant studies."  We read  the claim that up to 30 percent of research turns out to be wrong or consist of exaggerated results. 

We read about how badly published results fail to be replicated. We have a big boldface section header saying this:

"Replicating results is crucial. But scientists rarely do it."

We get this example

"The stats bear this out: A 2015 study looked at 83 highly cited studies that claimed to feature effective psychiatric treatments. Only 16 had ever been successfully replicated. Another 16 were contradicted by follow-up attempts, and 11 were found to have substantially smaller effects the second time around. Meanwhile, nearly half of the studies (40) had never been subject to replication at all."

We have this statement about misleading science journalism and misleading university press releases:

"Science journalism is often full of exaggerated, conflicting, or outright misleading claims...Sometimes bad stories are peddled by university press shops....Indeed, one review in BMJ found that one-third of university press releases contained either exaggerated claims of causation (when the study itself only suggested correlation), unwarranted implications about animal studies for people, or unfounded health advice....The 'toxic dynamic' of journalists, academic press offices, and scientists enabling one another to hype research can be tough to change, and many of our respondents pointed out that there were no easy fixes — though recognition was an important first step."

The long 2016 Vox article mentions some ways that this sorry state of science research could be improved.  But in the ten years since the article was published, there has been no improvement in the sorry state of science research. All the problems discussed in the 2016 article are still there, and still exist as badly as they existed in 2016.  There is no evidence that research scientists and science journalists are improving their dysfunctional and defective methods.  The many severe problems mentioned are only part of the problems that exist.  Many other severe problems in science research and science journalism are not mentioned in the Vox article, such as these:

(1) The tendency of scientific researchers to try to do research that supports prevailing dogmas of scientists, which are often groundless dogmas or poorly supported dogmas, rather than to do objective research that takes a "follow the evidence wherever it leads" approach. 

(2) The strong economic motivations that underlie misleading clickbait headlines, motivations such as the desire to produce page views that are profitable because of revenue-generating ads on such pages. 

(3) The use of way-too-small study group sizes in fields such as neuroscience, resulting mostly in unreliable "false alarm" results. 

(4) The use of poor methods of measurement in fields such as neuroscience, such as the widespread use of unreliable judgments of "freezing behavior" to judge fear or recall in rodents, rather than other more reliable methods. 

(5) A failure to follow a detailed blinding protocol.

(6) The extensive use of "keep torturing the data until it confesses" tactics, in which scientists fail to commit themselves to one straightforward method of gathering data and analyzing data, and instead act as if they had a license to endlessly play around with data, subjecting data to the most bizarre and convoluted arbitrary analysis pathways, that end up distorting and contorting the data gathered. 

What goes on in today's experimental neuroscience

When such problems exist in abundance, neuroscientists are largely engaging in a sham and a scam when they take federal money and pretend to be engaging in rigorous experimental science. 

Research science and science journalism are broke, and there is no sign that they are slowly mending themselves. 

A recent article on the Retraction Watch site is captured in the screen shot below. Notice the graph showing that the growth of fake or shoddy "paper mill" papers is stronger than the growth of regular scientific papers. 

So-called "paper mills" are for-profit companies that offer "editorial services" to scientists, which can range from writing much of a paper to writing an entire paper. The outputs of paper mills are very often fake papers and programmatically generated papers in which everything is written by computer programs, programs that steal most of their text from other papers. Such paper mills make heavy use of AI programs such as ChatGPT, which often give wrong answers or low-quality text sometimes called "AI slop." Someone can ask a program such as ChatGPT to generate a hypothetical paper sounding like the typical paper published in some line of research. The same person might then try to get the AI-generated paper published as a real paper. Or someone might ask a system like ChatGPT to write a summary of research on some narrow topic.  That person might then try to get the paper published as a "review article" or "systematic review" or "meta-analysis." Many scientists unwilling to get involved with shady paper mills are using AI systems such as ChatGPT to write much of the text of their papers. The integrity and credibility of the paper may be compromised by the use of AI-generated text not written by any mind understanding the topic discussed. 

As Exhibit A to back up the claim that the state of scientific research may be worsening, I may offer a recent article by Ross Andersen in The Atlantic, one entitled, "Science Is Drowning in AI Slop." We read this, referring to the "large-language models" used by so-called artificial intelligence or AI:

"Almost immediately after large language models went mainstream, manuscripts started pouring into [scientific] journal inboxes in unprecedented numbers. Some portion of this effect can be chalked up to AI’s ability to juice productivity, especially among non-English-speaking scientists who need help presenting their research. But ChatGPT and its ilk are also being used to give fraudulent or shoddy work a new veneer of plausibility, according to Mandy Hill, the managing director of academic publishing at Cambridge University Press & Assessment. That makes the task of sorting wheat from chaff much more time-consuming for editors and referees, and also more technically difficult."

Because of the AI slop problem, the state of science research may be even worse today than the very bad state described in the 2016 Vox article.