If you want to measure differences between groups, measure differences between groups.

Les Carter writes points to the article, Coming apart? Cultural distances in the United States over time, Marianne Bertrand and Emir Kamenica, which states:

There is a perception that cultural distances are growing, with a particular emphasis on increasing political polarization. . . .

Carter writes:

I am troubled by the inferences in the paper.

The authors state: “We define cultural distance in media consumption between the rich and the poor in a given year by our ability to predict whether an individual is rich or poor based on her media consumption that year. We use an analogous definition for the other three dimensions of culture (consumer behavior, attitudes, and time use) and other group memberships. We use a machine learning approach to determine how predictable group membership is from a set of variables in a given year. In particular, we use an ensemble method that combines predictions from three distinct approaches, namely elastic net, regression tree, and random forest (Mullainathan and Spiess 2017).” And come up with this:

This looks akin to ANOVA or a discriminant analysis. It seems to show (invalidated) predictability, but I can’t get from there to a measurement of trait differences among categories, especially as they include traits that are poorly predictive of category, namely time use. Is there reason to infer that they measure the degree of distinctiveness, or some kind of distance? Or is this analogous to the speaker I once witnessed using p values to rank effectiveness of various therapies?

My reply:

I took a look at the article and I too am unhappy with the indirect form of the analyses there. The questions asked by the authors are interesting, but if they want to study the differences in cultural behaviors of different groups, I’d rather see that comparison done directly, rather than this thing of using the behaviors to predict the group. I can see that the two questions are mathematically connected, but I find it confusing to use these indirect measures. When, in Red State Blue State, we were comparing the votes of upper and lower-income people, we just compared the votes of upper and lower-income people, we didn’t try to use votes to predict people’s income.

Again, the research conclusions in that paper could be correct, and I assume the authors put their data and code online so anyone can recover these results and then go on and do their own analyses. I just find it hard to say much from this indirect data analysis that’s been done so far.

P.S. I also noticed one little thing. The authors write, “We focus on the top and the bottom quartile (as opposed to, say, the top and the bottom half or the top and the bottom decile) to balance a desire to make the rich and the poor as different in their income as possible and the pragmatic need to keep our sample sizes sufficiently large.” That’s right! If you want to make simple comparisons (rather than running a regression), it’s a good move to throw out those middle cases. For more detail, see my paper with David Park, Splitting a predictor at the upper quarter or third and the lower quarter or third, which we wrote because we were doing this sort of comparison in Red State Blue State.

P.P.S. Please round those percentages in the tables. “71.4%,” indeed.