The Social Construction of Eager Community Mirages

People who believe untrue things often are convinced that their incorrect belief is based on evidence.  This can occur whenever there is some enthusiastic community of researchers very interested in gathering evidence in favor of such a belief. If the community of researchers is well-motivated and well-funded, it may be able to create an illusion of having a body of evidence establishing the dubious belief it is eager to prove.  We may call such a large group of researchers an eager community.  We may call the misleading body of evidence created by such a community an eager community mirage. 

The word "mirage" may refer to an optical illusion in which something appears to be in front of you, even though it isn't actually there (the classic example being some reflective material ahead of you that reflects the sky, fooling you into thinking there is a body of water ahead of you).  The word "mirage" can also refer to something that appears real but is illusory. 

Let me give a fictional example of an eager community mirage. Let us imagine a billionaire who dreams up a theory that the ghosts of dead animals live in the clouds, and that you might be able to see the ghost of your dead pet up in the sky. Having many millions to spead publicizing such an idea, we can imagine the billionaire selling many copies of some book that he wrote advancing this theory. 

Let us also imagine that the billionaire decides to spend millions of dollars trying to prove his theory. He might find thousands of people very interested in proving his strange theory, and might pay them each tens of thousands of dollars to try to prove his theory, by taking photographs of clouds in the sky, and looking for shapes that look like animals. 

Given such a large of researchers, getting such lavish funding, it would be likely that some type of superficially impressive "body of evidence" would accumulate. If the billionaire asked everyone of his thousands of well-funded researchers to send him a photo whenever they photographed a cloud that looked like an animal shape,  the billionaire would be able to accumulate a fairly nice little collection of clouds that looked like animals (particularly if each researcher had a financial incentive for each such photo sent to the billionaire). 

Would such a collection of photos be good evidence that dead animals become ghosts that live in the sky among the clouds? No, it would not be.  It would simply be the amount of evidence we would expect to get for such a hypothesis, given the very large community of eager researchers, and given the funding the billionaire had given them.  The body of evidence the billionaire would accumulate from such researchers would be an example of an eager community mirage.  Like a mirage, the illusion of good evidence would be largely based in reality.  The photos would not be faked, and would show real clouds. But the collection of such photos would not be robust evidence to prove the theory that the ghosts of dead animals rise up into the sky and live among the clouds. 

In the world of scientific academia, there exist various examples of bodies of evidence that appear to be mere eager community mirages. Such bodies of evidence can arise because there is a large community of many thousands of well-funded researchers eager to gather evidence for some particular dogma believed in by a belief community of scientists. 

Let us consider the body of evidence that is typically cited to support claims that the brain is the source of the human mind and the storage place of memories.  We do not find in such a body of evidence any "slam dunk" experiments or studies that provide "smoking gun" evidence in favor of such claims. Instead we find a whole bunch of studies providing far weaker evidence. 

Remarkably the standard for getting an experimental neuroscience paper published (and sold by some press release as being good evidence) is a very low standard, a very low hurdle to jump over. The convention is that you can get an experimental study published if your p-value is merely .05.  What is the p-value? It can be roughly thought of as the likelihood of you getting a particular result if your hypothesis of a causal effect is false. 

Let's imagine an example in a neuroscience experiment. Suppose I hypothesize that some region of the brain will light up more strongly than any other region under some particular example of mental activity. I then scan brains during this mental activity, and I get some result that I judge to have a p-value of .05.  That means that if there is actually no connection between that region of the brain and the mental activity I have tested, I should not have got such a result by chance in more than 1 in 20 experiments I did. 

A very important point is that the p-value is certainly not the likelihood about whether my result would show up if many experimenters were trying my experiment. It is merely something like the likelihood of me getting the result by chance on any particular time I tried the experiment. 

Now, is it anything like convincing evidence if I do some experiment getting such a p-value of .05? Certainly not. In fact, if I do the experiment twenty times, I should expect purely by chance to get such a result about 1 time in 20, even if my hypothesis about cause and effect is totally false. 

Now let us imagine a very large body of many thousands of well-funded neuroscience researchers. Altogether they have many hundreds of millions of dollars of funding, which each researcher can partially spend 30 weeks a year trying different experiments.  A study estimated there were about 300,000 neuroscience papers published in a ten-year period, about 30,000 per year. The actual number of neuroscience experiments done could easily be 100,000 or more per year, because of a "file drawer" by which null results are not even written up, or not published.  

How many results would we expect to get each year with a p-value of .05, purely by chance, even if brains do nothing to produce the human mind, and even if brains do not at all store memories?  Very many. In fact, we should expect to get thousands of such experiments producing a p-value of .05 or smaller, even if   brains do nothing to produce the human mind, and even if brains do not at all store memories. We also should expect to see hundreds of experiments with a more impressive p-value of only .001,  purely by chance, even if brains do nothing to produce the human mind, and even if brains do not at all store memories. Since tens of thousands of neuroscience experiments are being done around the world, we would expect that purely by chance hundreds of these experiments would produce results that had a chance probability of only about .001, even if no brain cause was producing the results.  We should also remember that scientists very often claim p-value results much more impressive than their observations warrant, as happened in the BICEP2 affair and the subsequently discredited  "phosphine on Venus" paper. 

What happens during the social construction of eager community mirages is that members of the eager community go searching for all of the results that best support the belief they want to believe in, and discuss these results in a single article or paper, often a scientific paper called a "review article." Gathered together, such results may seem impressive. But the appearance of some impressive reality is very often a mere mirage.  The results discussed may be merely exactly what we would expect to get by chance, given the size of such a research community, its eagerness to establish some particular result, and the number of trials that are being done.  

To give some examples, if there exists some large eager community desiring to prove some theory that the ghosts of animals live in clouds, and such a community is well funded by millions of dollars each year, we would expect that members of this community would spend many thousands of hours each year photographing clouds and looking for shapes that look like the ghosts of dead animals; and we would expect that every year some superficially impressive results would be produced by such a community.  But we would merely be seeing what we would expect to get by chance, even if the ghosts of dead animals don't live in clouds. Similarly, if there exists some large eager community of neuroscientists desiring to prove some theory that brains produce minds and that brains store memories, and such a community is well funded by billions of dollars each year, we would expect that members of this community would spend many thousands of hours each year doing experiments trying to show that brains produce minds and that brains store memories; and we would expect that every year some superficially impressive results would be produced by such a community.  But we would merely be seeing what we would expect to get by chance, even if brains do not produce minds and do not store memories. 

Defective or questionable research practices are a key factor facilitating the social construction of eager community mirages. The weaker the standards followed, the easier it will be for the eager community to socially construct the appearance it is trying to create. In experimental neuroscience we see such defective or questionable research practices very often. To give examples:

• Scientists know that the most reliable to do an experiment is to first state a hypothesis, how data will be gathered, and how data will be analyzed, using methods called "pre-registered studies" or "registered reports." But most experimental neuroscience studies do not follow such a standard, but instead follow a much less reliable technique, in which data is gathered, and then the experimenter is free to slice and dice the data in any way he wants, trying to prove any hypothesis he may dream up after collecting the data. 
• Because very many neuroscience observations are the kind of observations where subjective interpretations may be at play, a detailed and rigorous blinding protocol is an essential part of any reliable neuroscience experiment. But such a blinding protocol is rarely used, and in the minority of neuroscience experiments that claim to use blinding, the blinding will usually be only fragmentary and fractional. 
• Neuroscience experiments trying to measure fear in rodents can only do that reliably by measuring heart rate in such animals (which dramatically spikes when mice are afraid). But instead of using such a reliable technique, the most common practice in rodent experiments involving fear is to use an unreliable and subjective technique involving trying to judge so-called "freezing behavior."
• Brain scanning experiments typically present misleading visuals in which differences of less than 1% in brain activity are depicted in bright red in a brain diagram, creating the incorrect impression there was some big difference in activity in such a region. 
• A web site describing the reproducibility crisis in science mentions a person who was told of a neuroscience lab  "where the standard operating mode was to run a permutation analysis by iteratively excluding data points to find the most significant result," and quotes that person saying that there was little difference between such an approach and just making up data out of thin air. 

The neuroscientist community (very eager to prove dogmas that brains create minds and store memories) is only one example of eager communities in the world of scientific inquiry. Another such community is the origin-of-life research community, which for many decades has been eager to prove that life could have naturally originated from chance chemical reactions. 

A key element in the social construction of an eager community mirage may be biased interpretation of research results.  We have a gigantic example of this in the famous Miller-Urey experiment. In that experiment a small sealed glass apparatus was filled with a mixture of gases consisting of methane, ammonia and hydrogen, and subjected to continuous discharges of electricity for a week.  The result was some amino acids that formed at the bottom of the apparatus. For seventy years the eager origin-of-life research community has spread the groundless idea that such an experiment did something to show a likelihood of amino acids forming in the early Earth.  This claim never made any sense. Showing that some chemicals can form in a small sealed glass apparatus subject to continuous electricity discharge does nothing to show that such a formation would have occurred in the open atmosphere, both because gases and chemicals in the open atmosphere would have been many trillions of times more dispersed, and also because lightning in the atmosphere only occurs occasionally rather than continuously. But for 70 years the eager community of origin-of-life researchers has  misinterpreted the experiment as one showing that amino acids would have been common in the early Earth. 

Similar things happen in the neuroscientist community.  Scientists put whatever "spin" on their research results that most fit in with the belief dogmas they are eager to prove.  Such dubious or biased interpretations are endlessly repeated by other scientists eager to show that there is some evidence for some claim they want to believe in. 

I can give a little equation summarizing what I have discussed above:

Large community eager to prove some idea + lavish funding + weak research standards  + biased interpretation = occasional superficially persuasive results.

The "eager community mirage" arises when such occasional superficially persuasive results are collected from many years of effort by such a community. The result is something that may look like some body of evidence seeming to support the idea or dogma the community is eager to prove. But the result may be merely a mirage. 

A physical mirage does not stand up well to close inspection. On a hot road you may see in the distance something that looks like some water on the far horizon, but driving a hundred meters closer does not make that appearance seem more concrete. 

Similarly, socially-constructed eager community mirages do not stand up well to close inspection. The more closely we examine the techniques used to construct such mirages, the more likely we may be to realize that the body of evidence offered by the eager community to prove its favored beliefs is a mere mirage.