(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)
The other day I posted on a controversy in sociology where Aliya Saperstein and Andrew Penner analyzed data from the National Longitudinal Survey of Youth, coming to the conclusion that “that race is not a fixed characteristic of individuals but is flexible and continually negotiated in everyday interactions,” but then Lance Hannon and Robert DeFina argued that those claims were mistaken, the product of a data analysis that did not properly account for measurement error. I did not check out the details in either of the two sides’ analysis, but Hannon and DeFina’s criticism made sense to me, in that it did seem to me that measurement error could cause the artifacts of which they wrote.
As I wrote in my post, the story so far seems consistent with the common pattern in which a research team finds something interesting and then, rather than trying to shoot down their own idea, they try to protect it, running robustness studies with the goal of confirming their hypothesis, responding to criticisms defensively, and so forth. Just par for the course, and it’s a good thing that the journal Sociological Science was there to publish Hannon and DeFina’s article.
Anyway, we had some good blog discussion, including this detailed comment by Aaron Gullickson:
Given that I [Gullickson] am a co-author with Saperstein on another paper on racial fluidity (in this case switching between “black” and “mulatto in the 19th century US south), I have given some serious thought to what exactly is going on here to produce bias, since it is not particularly well spelled out by Hannon and LaFina. Based on simulations, I believe the potential for bias from reporting error exists in the models that control for prior racial identification (models 1-3 above) but not in the fixed effects model (model 4). Admittedly, the fixed effect estimate is smaller here and if you look at some of S&P’s other work, it tends to be less robust across different dependent variables and model specifications, but I think it is too early to start throwing out the results entirely. Its also difficult to know what proportion of switchers are actually data errors (although I have seen a draft of S&P’s AJS response and I think they have some clever ways of thinking about it, but I won’t steal their thunder).
Here is a replicable simulation in R outlining the source of the bias: http://pages.uoregon.edu/aarong/reportingerror.html
I don’t quite understand the “steal their thunder” thing, as Saperstein and Penner could post a preprint of their American Journal of Sociology response right now? No need to wait until formal publication, why not get the response out right away, I’d think.
Anyway, I sent a message to Hannon asking if he had any response to the above comment by Gullickson. Here’s how Hannon replied:
Gullickson seems to be making an argument that is consistent with our call for greater use of fixed-effects models in this literature. I don’t agree with his suggestion that accounting for measurement error in racial fluidity studies means a total focus on missing completely at random because “Anything else gets into the tricky question of what measurement error really means anyway on a variable with no real true response.” I think non-random measurement error matters too in this context. More generally, I hope that researchers in this area eventually move away from the idea that when an effect cannot be explained by random error, it must be due to their particular hypothesized causal mechanism (racial priming and the internalization of stereotypes). There are ways to more directly test their mechanism.
I’m happy that the discussion of this paper is centering on measurement issues. As I’ve written many times, I think the importance of careful measurement is underrated in statistics and in social science.
Also, Hannon said one thing I like so much I’ll repeat it right here:
More generally, I hope that researchers in this area eventually move away from the idea that when an effect cannot be explained by random error, it must be due to their particular hypothesized causal mechanism.
Yes. Rejection of the straw-man null should not be taken as acceptance of the favored alternative.
The post Racial classification sociology controversy update appeared first on Statistical Modeling, Causal Inference, and Social Science.
Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science
