The Rogues Gallery

Randi’s Challenge and Scientific Proof

I recently received the following e-mail asking a question about Randi’s Million Dollar Psychic Challenge:

Hi Skeptics,
I just thought I’d pass along this link I found on Fark. It’s a pro-psi article that slams Randi and his million dollar challenge. This in and of itself isn’t anything new (eh, they’re whining again, how cute), but one of the things that caught my eye was the claim that Randi’s challenge is a lot more stringent that what is usually scientifically required. In other words, they’re claiming that Randi’s challenge is harder to pass than to prove scientifically.
This doesn’t make sense to me. I’m sure there’s some misrepresentation on their part, but then again they have a pretty lengthy section on the probabilities required for various tests in Randi’s challenge, and how ridiculously high (allegedly) they are; something like 99%.
Anyway, I was hoping you guys could take a look at the link and let me know what you think. If nothing else, you can laugh at some paranormal special pleading. Love the show, keep it up.
-Jacob

Thanks for the question, Jacob. I have heard this complaint numerous times before, and have had several discussion about it with “believers.” The criticism stems from a misunderstanding of the nature of scientific “proof.” First, for some quick background, the Randi challenge is essentially that he will give 1 million dollars to anyone who can demonstrate under properly controlled conditions a paranormal ability. Con artists and true believers have come up with every excuse in the book as to why they fail or refuse to take the challenge. This one is about as silly as the rest – that it is harder to pass the challenge than to prove something scientifically. There are even those who argue that during the official tests for the challenge a standard P-value of 0.05 should be used. What this means is that, even in a world completely devoid of any paranormal phenomena, on average every 20th deluded boob or con artist would walk away with Randi’s million just by luck alone. Not gonna do it!

But what about the question of scientific proof? It is absurd to claim that all that is required is a single study with a 5% or even 1% chance of rejecting the null hypothesis for a completely new phenomenon to be accepted by the scientific community as probably real (which is what is really meant by “scientific proof”). Before something completely new becomes generally (and provisionally) accepted multiple studies need to show that the phenomenon is real. This includes exploring all reasonable alternative hypotheses. This means that before being accepted the cumulative probability of falsely rejecting the null hypothesis (sorry for the triple negative) is somewhere on the order of magnitude of a million to one against.

Of course there is no black and white acceptance or rejection is science. The more evidence there is for a claim the more it is treated as probably true. There is some fuzzy point beyond which it would be perverse not to consider a claim as an established scientific fact. Where that point is for a claim depends upon something we call prior probability – which is just an assessment of how likely a claim is to be true given all relevant prior scientific evidence. The more improbable a claim the more evidence the scientific community would require to accept it. If we apply this principle to paranormal claims, then Randi, if anything, is being very generous to the claimants.

Over at Science Based Medicine, Kimball Atwood has written a couple of nice posts discussing the role of prior probability in evaluating the medical literature. He convincingly argues that a Bayesian analysis that takes into consideration the prior probability is far better (especially when evaluating low probability claims) than a simple P-value. This type of analysis shows that when trying to establish a new phenomenon of nature, one that has a very low prior probability given what has been currently established by science, the requirement for new evidence to overturn existing scientific knowledge can be huge – far greater than what Randi would require to win the million dollars.

So once again the paranormal apologists should just quit their whining. Otherwise they just risk further exposing their relative ignorance of scientific methodology.