(This article was originally published at Econometrics Beat: Dave Giles' Blog, and syndicated at StatsBlogs.)
The statistical and econometric literature on the properties of "preliminary-test" (or "pre-test") estimation strategies is large and well established. These strategies arise when we proceed in a sequential manner when drawing inferences about parameters.
A simple example would be where we fit a regression model; test if a regressor is significant or not; and then either retain the model, or else remove the (insignificant) regressor and re-estimate the (simplified) model.
The theoretical literature associated with pre-testing is pretty complex. However, some of the basic messages arising from that literature can be illustrated quite simply. Let's look at the effect of "pre-testing" on the bias of the OLS regression estimator.
Consider the following bivariate regression model:
yi = β1x1i + β2x2i + ui ,
where x1 and x2 are non-random; and ui ~ i.i.d. N[0 , σ2] for all i.
Suppose that we adopt the following strategy:
1. Test to see if x2 is a statistically significant regressor.
2. If our test suggests that it is significant, then retain both x1 and x2 in the model.
3. If our test suggests that it is not, then drop x2 from the regression, and re-estimate the model keeping just x1 as the sole regressor.
Let's focus on the estimator of β1 that's actually associated with this strategy. If we stop at step 2, denote the (unrestricted) OLS estimator (MLE) as b1U. If we stop at step 3, denote the (restricted) OLS estimator (MLE) as b1R.
The null hypothesis that we'll be testing is H0: β2 = 0, and the alternative hypothesis is HA: β2 > 0 (although nothing of substance changes if we adopt a two-sided HA).
Given our assumptions about the error term in the model, the obvious (and UMP) test will be a t-test. Denote the associated test statistic by "t2", and let tc(α) be the associated critical value when the chosen significance level is α.
So, the"pre-test" estimator of β1 is of the following form:
β1* = b1U ; if t2 > tc(α)
β1* = b1R ; if t2 ≤ tc(α)
One final bit of notation will be particularly helpful in exposing the result that I want to illustrate.
Let I[A](x) be an "indicator function" that takes the value "1" if the random variable, x, lies in the interval A, and is zero otherwise. Let A' be the "complement" to the interval A. So, if x does not lie in A, it lies in A', and vice versa.
Note that I[A'](x) = 1 - I[A](x); and I[A](x)I[A'](x) = 0 , for any A and x.
Also, I[A](x) is a binary random variable, as it is a function of x.
Letting x = t2, we can write our pre-test estimator of β1 as:
β1* = I(tc(α) , ∞)(t2) b1U + I[0 , tc(α)](t2) b1R
= {1 - I[0 , tc(α)](t2)} b1U + I[0 , tc(α)](t2) b1R
= b1U + I[0 , tc(α)](t2) (b1R - b1U) .
Let's think about the biases of b1U and b1R, under the assumptions associated with our model.
- b1U is always unbiased, whether H0 is true or false. (Over-fitting the model will not introduce a bias in the OLS estimator.)
- b1R is unbiased if H0 is true, but it is biased if H0 is false. (Omitting a relevant regressor biases the OLS estimator.)
With this in mind, consider the expected value of β1*:
E[β1*] = E[b1U] + E{I[0 ,tc(α)](t2) (b1R - b1U)}
= β1 + E{I[0 , tc(α)](t2) (b1R - b1U)}.
The second term on the right of the last equation is typically non-zero, and it's going to be messy. That's because t2 (and hence the indicator function) is not independent of (b1R - b1U).
This second term is the bias in the (pre-test) estimator, β1*.
So, even without going to all of the trouble to evaluate the bias exactly - and it actually is quite a lot of trouble, even for this very simple model - we can see that pre-testing in the context of OLS estimation will typically introduce a bias.
A sufficient condition for this will be met if b1R = b1U, and this in turn will be satisfied if the OLS estimator of β2 in the original two-regressor model happens to yield a point estimate of exactly zero. Putting this extreme case to one side, the pre-test estimator of β1 will be biased.
The basic result demonstrated here extends fully to the case where we have a multiple regression model, and instead of testing a single "zero" restriction we test the validity of a set of independent linear restrictions on the coefficient vector, using an F-test.
Further, although pre-testing generally biases or regression estimator, it also has an impact on the latter's efficiency. In other words, it impacts on the estimator's MSE. This suggests the following question: "Is pre-testing necessarily a 'bad' strategy?"
I'll take this up in more detail in a subsequent post, but in the meantime you might check out my earlier related posts (here and here), and the survey material on pre-testing in Giles and Giles (1993).
Reference
Giles, J. A. and D. E. A. Giles, 1993, “Pre-test estimation and testing in econometrics: Recent developments”, Journal of Economic Surveys, 7, 145-197.
Please comment on the article here: Econometrics Beat: Dave Giles' Blog
