(This article was originally published at Econometrics Beat: Dave Giles' Blog, and syndicated at StatsBlogs.)
Some time ago, Haynes Goddard emailed me suggesting that I post something about step-wise regression. He also put me on an interesting paper by Peter Flom and David Cassell.
Many statistical and econometrics packages include stepwise regression. I wish they didn't! Here's why.
Flom and Cassell list several "strikes" against step-wise regression:1. R2 values are biased high.
2. The F and χ2 test statistics do not have the claimed distribution.
3. The standard errors of the parameter estimates are too small.
4. Consequently, the confidence intervals around the parameter estimates are too narrow.
5. p-values are too low, due to multiple comparisons, and are difficult to correct.
6. Parameter estimates are biased high in absolute value.
7. Collinearity problems are exacerbated.
I've quoted the authors directly - I'd have expressed some of these points a little differently, but I'm sure you get their drift.
The crux of the matter is this. Step-wise regression involves a search for a model specification based on the apparent significance (or otherwise) of various covariates. It's an example of model simplification, rather than model specification. The end result may make little economic sense.
Searches of this type involve fundamental "pre-testing" issues, unless it happens to be the case that the successive tests that are used are all statistically independent. This independence is absent with conventional step-wise regresssion procedures. When we test the significance of one potential regressor (regardless of the outcome), this affects the properties of the next (and all subsequent) tests that we perform. The true significance levels of the subsequent tests are not what we nominally assign them to be. Moreover, the extent to which these values are distorted depends on the true (unknown) values of the parameters in a very complicated way.
In short, the sequential testing associated with step-wise regression is fatally flawed.
In addition, the properties of the estimates of the parameters are no longer what we might think they are. For example, after even a single pre-test, the OLS estimator loses its usual property of unbiasedness. And things just get worse and worse as we continue with more testing.
If you want a (now out-of-date) survey of the pre-testing literature, with an econometrics slant, then take a look at Giles and Giles (1993).
One package that I use a lot is EViews. Although it has a facility for both forward and reverse step-wise regression, the documentation includes the following statement:
"The p-values listed in the final regression output and all subsequent testing procedures do not account for the regressions that were run during the selection process. One should take care to interpret results accordingly.
Invalid inference is but one of the reasons that stepwise regression and other variable selection methods have a large number of critics amongst statisticians. Other problems include an upwardly biased final R-squared, possibly upwardly biased coefficient estimates, and narrow confidence intervals. It is also often pointed out that the selection methods themselves use statistics that do not account for the selection process."
I totally agree with all of this, and the team at EViews should be applauded for including this warning.
As econometricians, we're concerned with more than just a parsimonious model. Usually, we're very interested in the economic implications of our model specification and the quality of our parameter estimates.
Unfortunately, with the advent of "big data", and situations where there are far more potential covariates than there are data-points, techniques such as step-wise regression seem to be back in vogue again. Grrrrr!!!!!
Unfortunately, with the advent of "big data", and situations where there are far more potential covariates than there are data-points, techniques such as step-wise regression seem to be back in vogue again. Grrrrr!!!!!
Reference
P. L. Flom and D. L. Cassell, 2007. Stopping stepwise: Why stepwise and similar selection methods are bad, and what you should use. NESUG 2007 Proceedings.
Giles, J. A. andD. E. A. Giles, 1993. Pre-test estimation and testing in econometrics: Recent developments. Journal of Economic Surveys, 7, 145-197.
Giles, J. A. andD. E. A. Giles, 1993. Pre-test estimation and testing in econometrics: Recent developments. Journal of Economic Surveys, 7, 145-197.
Please comment on the article here: Econometrics Beat: Dave Giles' Blog
