(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)
John Cook writes:
Some physicists say that you should always have an order-of-magnitude idea of what a result will be before you calculate it. This implies a belief that such estimates are usually possible, and that they provide a sanity check for calculations. And that’s true in physics, at least in mechanics. In probability, however, it is quite common for even an expert’s intuition to be way off.
I agree with Cook’s general message but I’d say it slightly differently. I’d say that even many experts often have no intuition at all when it comes to probability, which will lead them to miss huge conceptual errors in their calculations.
The example that comes to mind is the largely atrocious literature in political science and economics on the probability of a decisive vote. A paper was published in a political science journal giving the probability of a tied vote in a presidential election as something like 10^-90. Talk about innumeracy! The calculation, of course (I say “of course” because if you are a statistician you will likely know what is coming) was based on the binomial distribution with known p. For example, Obama got something like 52% of the vote, so if you take n=130 million and p=0.52 and figure out the probability of an exact tie, you can work out the formula etc etc.
From empirical grounds that 10^-90 thing is ludicrous. You can easily get an order-of-magnitude estimate by looking at the empirical probability, based on recent elections, that the vote margin will be within 2 million votes (say) and then dividing by 2 million to get the probability of it being a tie or one vote from a tie.
The funny thing—and I think this is a case for various bad numbers that get out there—is that this 10^-90 has no intuition behind it, it’s just the product of a mindlessly applied formula (because everyone “knows” that you use the binomial distribution to calculate the probability of k heads in n coin flips). But it’s bad intuition that allows people to accept that number without screaming. A serious political science journal wouldn’t accept a claim that there were 10^90 people in some obscure country, or that some person was 10^90 feet tall. But intuitions about probabilities are weak, even among the sort of quantitatively-trained researchers who know about the binomial distribution.
P.S. The point of this post is not to bang on the people who made this particular mistake but rather to use this as an example to illustrate the widespread lack of intuition about orders of magnitudes of probability, which is relevant to John Cook’s point regarding statistical thinking and communication.
Another example is business-school prof Reid Hastie’s apparent belief that “the probability that a massive flood will occur sometime in the next year and drown more than 1,000 Americans” is more than 20%. 20% sounds like a low number, low enough that Hastie didn’t consider that such floods have been extremely rare in American history. (Even Katrina drowned only 387 people, according to this source which I found by googling Katrina drownings.) This is not to disparage the importance of preparing for floods; even if the probability is only 1%, it’s still makes sense to do what we can to mitigate the risks. My point here is just that probabilities are hard to think about. It’s Gigerenzer’s point.
To continue with the Gigerenzer idea, one way to get a grip on the probability of a tied election is to ask a question like, what is the probability that an election is determined by less than 100,000 votes in a decisive state. That’s happened at least once. (In 2000, Gore won Florida by only 20-30,000 votes.) The probability of an exact tie is of the order of magnitude of 10^(-5) times the probability of an election being decided by less than 100,000 votes.
See, that wasn’t so hard! Gigerenzering wins again.
The post Common sense and statistics appeared first on Statistical Modeling, Causal Inference, and Social Science.
Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science
