No, I don’t buy that claim that Fox news is shifting the vote by 6 percentage points

Tyler Cowen writes:

This is only one estimate, from Gregory J. Martin and Ali Yurukoglu, but nonetheless it is backed by a plausible identification stragegy and this is very interesting research:

We find that in a hypothetical world without Fox News but with no other changes, the Republican vote share in the 2000 election would have been about half a percentage point lower. By 2008, the effect of there being no Fox News rises to more than six percentage points – a result of the channel’s increasing viewership and increasingly conservative slant over this period.

I am skeptical, for reasons discussed in this post from a couple years ago. It’s a problem of extrapolation.

Here’s what I wrote in 2016:

[Martin and Yurukoglu] found that Fox’s impact took off during the decade that followed: It increased Republican vote share by more than 3.5 percentage points in the 2004 election and more than 6 percentage points by 2008. . . .

Where did that 6-percentage-point estimate come from? I clicked through to Martin and Yurukoglu’s article, which begins as follows:

We measure the persuasive effects of slanted news and tastes for like-minded news, exploiting cable channel positions as exogenous shifters of cable news viewership. Channel positions do not correlate with demographics that predict viewership and voting, nor with local satellite viewership. We estimate that Fox News increases Republican vote shares by 0.3 points among viewers induced into watching 2.5 additional minutes per week by variation in position. We then estimate a model of voters who select into watching slanted news, and whose ideologies evolve as a result. We quantitatively assess media-driven polarization, and simulate alternative ideological slanting of news channels.

That 0.3 percentage points is a lot less than 6 percentage points. You can, however, get pretty close if you scale up from 2.5 minutes per week to 1 hour per week: 0.003*60/2.5 = 0.072, and maybe the difference between that and 0.006 can be explained by rounding. (For example, if the reported 0.3 is a rounded 0.026, then you get 0.026*60/2.5 = 0.0624, which rounds to 0.06.)

I then searched the Martin and Yurukoglu paper for the 6-percentage-point estimate. I found it in table 14, which reports the estimate that were Fox News to have disappeared in 2008, the Republican candidate would’ve lost 6.3 percentage points of the vote.

President Obama beat John McCain in 2008 by the margin 53 percent to 46 percent, so according to this model, had Fox News that year disappeared (or, I suppose, switched to a politically neutral format), Obama’s electoral margin would’ve been a Reaganesque 59 percent to 40 percent.

I don’t believe it.

What, then, went wrong in this analysis? Lots of little things, many of which are indeed mentioned by Martin and Yurukoglu. First is the extrapolation from 2.5 minutes per week to an hour per week, which assumes a linear effect (no diminishing returns) and also which takes the estimate far from what can be seen directly from the data.

Second, there’s the assumption that a change in Fox News would happen in a vacuum.

Third, there’s uncertainty. I’ll take the authors’ word that these estimates are statistically significant—that is, that one would not see such a pattern from chance alone—but there’s still going to be a lot of variation in these numbers. And in such settings, effect sizes tend to be overestimated. Estimates near zero are discarded and high estimates are reported. We call this the “statistical significance filter.” . . .

I think Martin and Yurukoglu’s paper is interesting and I think they’re admirably careful both in their presentation and their summary. They very appropriately gave their empirical estimate of 0.03 percentage points in the abstract, putting the larger claims deep in the paper with lots of qualifiers. [In their report of this study in Slate] Fisman and Prat were also careful to report that their conclusions were based on a model.

Still, something went wrong in the presentation, because the Slate article reads like this out-of-control 6-percentage-point extrapolation is real. . . .

For further background, see this 2014 paper by political scientists Dan Hopkins and Jonathan Ladd, who analyze data from a 2000 pre-election poll and find a positive effect of Fox News on support for George W. Bush, but “only on the vote intentions of Republicans and pure independents.” In summarizing this study, Hopkins writes that media influence “fosters political polarization. For Republicans and pure independents, Fox News access in 2000 reinforced GOP loyalties.”

I recognize that Cowen expressed mild skepticism in his post (“This is only one estimate”) but I don’t think he was skeptical enough! All the identification in the world doesn’t resolve the problem of extrapolation.

Again, I say this not to criticize the work of Martin and Yurukoglu. To say that we have to be careful about overintepreting a research result is not to say that the research is bad.