Phosphine is a gas mainly produced on Earth by living things. On September 14, 2020, a scientific paper entitled "Phosphine gas in the cloud decks of Venus" claimed to have detected the "apparent presence" of phosphine in the atmosphere of Venus. The lead author was Jane S. Greaves. The claimed abundance level was very tiny, only about 10 parts per billion. Following their long-standing tendency to hype like crazy any story that may serve as clickbait to produce more page views and advertising revenue, a host of web sites began proclaiming that life or a sign of life had been discovered at Venus.
I may have been the first one in the blogosphere to make a substantive criticism of this claim, since I published on the next day a post entitled "No, They Haven't Detected Life at Venus," in which I cited several reasons for doubting the idea that the paper had provided any evidence of life at Venus. I said on September 15, 2020 that an alternate explanation was that "an error in interpretation could have occurred in spectral data that is hard-to-interpret because of overlapping signals from a variety of different gases in the atmosphere of Venus," and that such a possibility was "not very unlikely." I also pointed out the substantial chance that merely geological processes could have produced phosphine, and pointed out the lack of any plausible scenario for life on Venus, given the incredibly hostile conditions both on its hot-enough-to-melt-lead surface and its clouds (having almost no water but lots of sulfuric acid vapor).
Days after my post, we began seeing on some sites such as www.liveScience.com and the National Geographic web site some articles (such as this one) questioning whether any sign of life had really been discovered on Venus. On October 19, 2020 there appeared a scientific paper with the title "Re-analysis of the 267-GHz ALMA observations of Venus: No statistically significant detection of phosphine." The paper doesn't merely question whether evidence of life on Venus has been found. The paper tells us that no robust evidence of phosphine has been discovered in the atmosphere of Venus.
The five authors of the scientific paper provide a critique of the September 14, 2020 paper claiming evidence for phosphine in the atmosphere of Venus, saying that the paper used a dubious statistical technique that "leads to spurious results." The five authors state that the data actually provides "no statistical evidence for phosphine in the atmosphere of Venus."
Four additional recent papers (published after the widely discussed "phosphine at Venus" paper) said that there is no phosphine in the atmosphere of Venus. One paper by a single author states, "There is thus no significant evidence for phosphine absorption in the JCMT Venus spectra." Another paper with many co-authors is entitled, "No phosphine in the atmosphere of Venus." A third paper states there is "no statistical evidence for phosphine in the atmosphere of Venus." Another paper says, "These findings, along with the recent papers by Encrenaz et al. (2020), Snellen et al. (2020), Lincowski et al. (2020), and Villanueva et al. (2020) undermine the reported detection of PH3 [phosphine] by Greaves et al. (2020a,b) and its possible biogenic origin." A news account of this paper says, "The team concluded that what the scientists probably saw was just sulfur dioxide, which is a common gas around Venus and would not indicate the possible presence of life."
What was going on in the Greaves paper claiming phosphine on Venus seems to have been pareidolia fueled by wishful thinking. Scientists looked at some noisy borderline data produced at the limits of observations, and interpreted the data in a way so that they could claim something they were eagerly hoping to find. Pareidolia typically occurs when someone looks at noisy, hard-to-interpret frequently-changing data, and then claims to see some important thing that is not really there. The example of someone claiming to have found a Jesus image in his toast is an example of pareidolia. Another example of pareidolia is shown below:
A few years later we had another case resembling the subsequently-discredited report of detecting phosphine on Venus. It was another case of a scientist claiming to have found something of great significance in the field of astrobiology, the search for life in outer space. In 2023 Nikku Madhusudhan and four other scientists created quite a stir. They authored a paper entitled "Carbon-bearing Molecules in a Possible Hycean Atmosphere." Researching a planet called K2-18 b revolving around another star, the paper claimed to have found "potential signs of dimethyl sulfide (DMS), which has been predicted to be an observable biomarker in Hycean worlds." The term "Hycean worlds" refers to planets in other solar systems that may be entirely covered by an ocean. The term "biomarker" refers to something that may be a sign of life. A very simple compound, dimethyl sulfide is not any type of building block of life. But on Earth dimethyl sulfide is sometimes produced by life.
But there were some reasons why the attempt to insinuate a biomarker was very dubious. One reason was that the claims about "potential signs of dimethyl sulfide" was a kind of "reading tea leaves" affair, in which scientists were analyzing the faintest of faint signals, rather like someone squinting at something on the horizon miles away. That type of observation offers plenty of opportunity to see what you want to see, by interpreting marginal hard-to-interpret just-barely-detectable data in some way that fits your cherished desires, rather than interpreting that data in a hundred other ways.
Then there is the fact that when scientists do observations like this, they are picking up signals from many different chemical sources, with the signals being all mixed up. It's a recipe for false alarms, rather like someone in a very crowded high school cafeteria trying to listen to what someone at a different cafeteria table far away is saying.
Despite the paper's failure to detect water, and its weak mention of a mere mention of "potential signs of dimethyl sulfide," the world's "give us an inch and we'll take a mile" science news press began publishing a flood of misleading stories falsely claiming that some promising sign of life had been found. After the "sugar rush" of this flood of misleading stories, other scientists got busy examining the data on the distant planet K2-18 b, to see whether there was any decent evidence for dimethyl sulfide. In 2024 scientists produced a paper arguing that K2-18 b was not a "Hycaean" planet covered by an ocean, but instead a gas planet like Neptune with no ocean. The paper was "JWST Observations of K2-18b Can Be Explained by a Gas-rich Mini-Neptune with No Habitable Surface" authored by Nicholas F. Wogan and others.
Then in early 2025 there was published the paper "A Comprehensive Reanalysis of K2-18 b's JWST NIRISS+NIRSpec Transmission Spectrum." It reanalyzed the data on K2-18 b and says "we find no statistically significant or reliable evidence for CO2 or DMS [dimethyl sulfide]." The paper had 16 authors, as compared to only five authors of Madhusudhan's paper. The 16 authors had found that Madhusudhan's claims about dimethyl sulfide at K2-18 b were unfounded.
In April 2025 Madhusudhan released a new paper, based on some new observations. He claimed to have found stronger evidence for dimethyl sulfide at the planet K2-18 b. For a few days the world's science news sites spread a Madhusudhan-crafted narrative that the "strongest evidence yet for extraterrestrial life" had been found. But Madhusudhan's claims were soon shot down by other scientists. An An Ars Technica article soon appeared entitled "Skepticism greets claims of a possible biosignature on a distant world." We read this:
One extremely common way for a scientist to conjure up a phantasm is to use too small a sample size or too small a study group size, which tends to result in false alarms. Scientists know how they can avoid this sin: by doing a sample size calculation to determine the minimum study group size needed to produce a moderately persuasive result, and to only use study groups with such a size. But a large fraction of scientific studies (particularly animal neuroscience studies) fail to include such a calculation, and fail to have adequate study group sizes.
Another way for a scientist to conjure up a phantasm is to prune or filter his data until the desired thing seems to appear. It might be that when considering his full data set, there will seem to be no evidence of some thing (call it X) that the scientist wants the data to show evidence for. But the scientist can prune the data at its beginning or end, until some evidence of X seems to show up. For example, if there were 4 weeks of data collection, the scientist can just get rid of the week 1 data or the week 4 data, or maybe both weeks of data. Or, the scientist can apply some "data filter" which gets rid of certain data points, until some evidence of X seems to show up. The decision to apply such a data filter can often be rationalized in various ways, to make it sound like some "quality filter" excluding "bad data" or "outlier data."
Another way for a scientist to conjure up a phantasm is to collect data in a biased way that will maximize the chance that the desired result will appear. I will give a hypothetical example. Let us imagine that you are a scientist who wants to show that rainy days in New York City cause a higher chance of 300-point drops in a stock market indicator such as the Dow Jones Industrial Average. You might begin recording daily stock market results on a rainy day in which there was a 300-point drop in the stock market. You might then continue to record daily results, and conveniently end your data collection on a rainy day in which there was more than a 300-point drop in the stock market. Given such convenient start and stop points of your data collection, you may well be able to write up a "statistically significant" correlation between rainy days in New York and 300-point drops in the stock market. But if you had resolved beforehand to start collecting data on some day 14 days in the future, and continue collecting data for exactly 100 days, then the desired result would probably not show up.
Once so-called "raw data" has been collected, there are 1001 ways to "massage" the data before it is analyzed, some of which may make sense and some of which are dubious. A scientist can produce all kinds of rationalizations for particular data exclusions and data inclusions and use of data averages or "data smoothing" and use of "weighted averages" that may have been used, which can have a huge effect on whether some illusory phantasm shows up.
Moving from the topic of data collection to data analysis, there are innumerable ways in which a scientist can conjure up phantasms by some kind of data analysis that isn't quite right. Don't be reassured when some science paper claims that it uses some kind of "standard software" for data analysis. There is almost always no such thing as a "standard analysis" of data. There are standard software tools used for data analysis, but such tools can be used in a million valid ways, and a million dubious ways. An example is Microsoft Excel, the leading spreadsheet program. There are a million bad ways to use it, as well as a million good ways.
When neuroscientists attempt to judge whether a rodent recalled by using the faulty "freezing behavior" method and manual inspection of how many seconds the rodent was immobile (within some arbitrarily chosen interval that can be anything between 30 seconds and five minutes), it's pretty much the perfect recipe for "see whatever you want to see" analysis. And when such analysis is done by people who are not blind to whether the observed rodents are in the experimental group or the control group, you have a likelihood of "see whatever you are hoping to see." Blinding fails to occur in most cognitive neuroscience studies, and almost never do we get a detailed statement giving us confidence that an effective protocol for blinding occurred. As for pre-registration (which can reduce some of the problems discussed in this post), a neuroscientist recently confessed it is rare in neuroscience.
A science paper may try to reassure you that it used some standard software for doing some type of data analysis (such as measuring brain scan data or trying to measure "freezing behavior" in mice). But there are always countless different ways to use such software, some good and some bad. Every software program has program settings or startup options or menu options that allow you to customize how the program is used. Software programmers usually ask "how can I give the user the freedom to do exactly what he wants," and almost never ask "how I can make it so that there's no way to use the software in a stupid way."
Nowadays neuroscientists often use "roll your own" customized computer programming to operate on gathered data in a way we can describe as "keep torturing the data until it confesses." AI programs make it easier than ever to create such software, which is often of low quality.
Another way a scientist can conjure up phantasms is by failing to do a preregistered study that announces an experiment will be testing one very specific hypothesis, and going on a kind of "fishing expedition" within his analytic activity. For example, let's imagine the scientist does brain scans looking for some correlation between some behavior (or some aspect of thinking) and activity in some tiny brain region, some particular hundredth of a brain. By failing to limit himself to checking one small specific part of the brain corresponding to a previously declared hypothesis, and giving himself the freedom to check any of 100 small parts of the brain, he will have a good chance of finding some tiny region that weakly correlates with the behavior or aspect of mentality. That's simply because given 100 parameters that show random variations, and the freedom to check any of them, it's easy to get something that looks like a slight correlation, even if only chance and not causation is involved. The name sometimes given for this procedural sin is HARKing, which stands for Hypothesizing After Results are Known.
Scientists have a hundred ways to conjure up illusory phantasms, and once a phantasm has been conjured up, there are many tricks by which the phantasm may be made to seem like something real, such as the use of complex charts and thick jargon which make the problematic presentation seem very scientific. All in all, we may say that the power of scientists to give you an impression of the reality of something illusory is comparable to the similar power of Hollywood's CGI special effects wizards.
Very much of the more interesting-sounding neuroscience research results are examples of "seeing what was not there" pareidolia, in which some false alarm is conjured up using one of the techniques discussed above. There is no actual evidence for non-genetic representations in the brain. But neuroscientists often claim to have found faint traces of such things. Such neuroscientists are doing work similar to the false alarm generation work of Greaves, Madhusudhan and Loeb.
Similarly, if there was a large community of 10,000 ardent fairy researchers who were funded with billions of dollars, we would probably occasionally see superficially persuasive papers offering evidence for fairies. For example, with such an army of researchers, and so much money to spend, there might be occasional infrared heat signature studies suggesting anomalous little blobs of heat floating about that might be interpreted as fairies. The researchers would be helped by the research rule that says, "Torture the data sufficiently, and it will confess to almost anything."
No comments:
Post a Comment