(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)
Nadia Hassan writes:
Nate Silver did a review of pre-election predictions from forecasting models in 2012. The overall results were not great, but many scholars noted that some models seemed to do quite well. You mentioned that you were interested in how top-notch models fare.
Nate agreed that some were better, but he raised the question of lucky vs. good with forecasters:
“Some people beat Vegas at roulette on any given evening. Some investors beat the stock market in any given month/quarter/year, and yet there is (relatively) little evidence of persistent stock-picking skill, etc, etc.”The other thing is you did a paper with Wang on the limits of predictive accuracy. Many election models are linear regressions, but the point seems pertinent.
Election forecasting is seen by some as a valuable opportunity social science theories over time. It does seem like one can go wrong by just comparing pre-election forecasts to outcomes. How can one examine predictions sensibly considering these issues?
My reply: One way to increase N here is to look at state-by-state predictions. Here it makes sense to look at predictions for each state relative to the national average, rather than just looking at the raw prediction. To put it another way: suppose the state-level outcomes are y_1,…,y_50, and the national popular vote outcome is y_usa (a weighted average of the 50 y_j’s). Then you should evaluate the national prediction by comparing to y_usa, and you should evaluate state predictions of y_j – y_usa for each j. Otherwise you’re kinda double counting the national election and you’re not really evaluating different aspects of the prediction. You can also look at predictions of local elections, congressional elections, etc.
And always evaluate predictions on vote proportions, not just win/loss. That’s something I’ve been saying for a long long time (for example see this book review from 1993). To evaluate predictions based on win/loss is to just throw away information.
The post Evaluating election forecasts appeared first on Statistical Modeling, Causal Inference, and Social Science.
Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science
