Clarity about the Counterfactual when Discussing "Campaign Effects"

July 17, 2012

(This article was originally published at Carlisle Rainey » Methods/Statistics, and syndicated at StatsBlogs.)

For a while now, political scientists have been trying to convince journalists that campaigns don't matter much or they matter in subtle ways, but journalists don't seem to get it, so political scientists continue trying to persuade. Judging by longevity of the dispute, journalists find this notion that campaigns don't matter much hard to swallow. I find it hard to swallow as well.

I think the confusion stems at least partly from differences in political scientists' and journalists' training. Political scientists are trained to think about "effects" in a very precise and non-obvious manner. Unfortunately, this thinking is often implicit and political scientists fail to communicate their reasoning to journalists. This leads to a good deal of discussion that seems to accomplish nothing. For example, see here, here, here, here, here, here, here, here, herehere, here, herehere, here, herehere, here, herehere, here, herehere, and here, just to take a few examples.

I think that journalists' intuition that campaigns have large effects is correct. I also think that political scientists' research suggesting that campaigns have small effects is correct. How can campaigns have both small and large effects? I think that journalists and political scientists are using different counterfactuals to define "effect."

I think that confusion about the counterfactual leads to much of
the disagreement and confusion about the size of campaign effects.


One cannot think about "effects" without implicitly reasoning about counterfactuals. Whether we realize it or not, counterfactuals define effects. Anytime we discuss an effect, we are referencing counterfactuals. If we are arguing about the size of effects, but using different counterfactuals, then we can expect a lot of difficulty reaching agreement.

What are counterfactuals?

Counterfactuals are a type of reasoning in which one asks, "How would the outcome have been different if I changed one particular aspect of a process? For example, we might ask how the vote shares would have changed if McCain had chosen Romney as his VP rather than Palin.

Counterfactual reasoning requires imagining two worlds or histories. The first world is typically the one we actually observed or things as they actually occurred. We create the second world by simply changing a single interesting feature of the observed world, perhaps McCain's VP choice. We then try to figure out what would have happened in the second world and compare it to the world that actually occurred. If the vote shares are similar, then we say that McCain's VP choice didn't matter much. If they are very different, the we would say that McCain's VP choice mattered.

Political scientists have a technical toolkit for assessing "what if" questions, but we don't need those tools here. We can just use imagination to develop plausible outcomes using educated guesses.

The Importance of Explicitly Considering the Counterfactual

Anytime an analyst discusses an "effect," it is a good idea to probe the counterfactual underlying the effect. Indeed, the counter-factual defines the effect--without understanding the counter-factual, one cannot understand the effect. Fortunately, political scientists are trained to think in this manner. Unfortunately, many do so implicitly, leaving journalists in the dark.

I contend that political scientists implicitly use unexpected counterfactuals
and this leaves journalists in the dark.

I think that political scientists' counterfactuals can differ from journalists' in two important ways. First, political scientists tend to discuss the effects of small changes in campaigns, while journalists tend to imagine big changes. Second, political scientists construct counterfactuals in which campaigns are responding to each other and cancelling out, while journalists tend to hold one campaign constant and vary the other.

Look at large or small changes?

Suppose that we observe an election, in which John McCain had a gaffe, mistakenly claiming that "the fundamentals of the economy are strong." He goes on to lose that election to Barack Obama 46% to 53%. What is the effect of his gaffe?

Well, to determine the effect of the gaffe, we need to know the election outcome in a hypothetical world in which John McCain did not have the gaffe. We just change one factor in this hypothetical world--remove the gaffe--and then let the campaigns play out. What would be the election outcome in this counter-factual? Suppose it was 47% to 52%. Then we can say that the effect of the gaffe was to lower McCain's vote share by 1 percentage point, raise Obama's vote share by 1 percentage point, and increase Obama's margin of victory by 2 percentage points.

One percentage point might underestimate or overestimate the effect, but I think we can be confident that the effect of the gaffe was small and that 1 percentage point is not too far off.

Does this mean that campaigns don't matter? I don't think so. When I hear someone claim that campaigns don't matter, I think of an entirely different counterfactual, and I think journalists do as well.

I imagine a much different counterfactual. Suppose we observed the world in which John McCain campaigned and compare it to a hypothetical world in which John McCain did not campaign. He did not mobilize voters, run ads, make speechs, participate in the debates, respond to Obama's criticisms, or have anyone do these things on his behalf. What would the election outcome look like in this hypothetical counterfactual world? I certainly don't know, but I'm guessing something like Obama wins 75% to 25%? In any case, it would be a landslide loss for McCain.

Using this counterfactual, we would say that McCain's campaign matters--a lot.

Political scientists study small changes

Political scientists use counterfactual that fall within the range of typical data. They don't muse about hypothetical worlds in which John McCain doesn't campaign at all. Instead, they try to figure out what would happen in plausible counterfactuals, such as a world in which John McCain ran a few more or a few less negative ads. Roughly, political scientists try to determine the effects of observed variation across campaigns, but presidential campaigns don't vary that much. We observe tiny differences in personality, but candidates who make it through tough primaries tend to be fairly likable. We observe tiny differences in strategy, but professional and thorough campaigns tend to adopt similar strategies. We observe some variation in resources, but presidential warchests tend to be similar. Political scientists suggest that campaigns much don't matter because campaigns don't vary much.

Conclusion: The size of campaign effects depends on the size of the changes in the counterfactual.

Application for Political Scientists: When claiming that campaigns don't matter much, be clear that you are talking about tiny changes in campaigns.

Application for Journalists: Understand that political scientists argue for tiny campaign effects because they are studying tiny changes in campaigns.

Hold the other campaign constant or allow it to respond?

Suppose that in 2008 Barack Obama decided to launch a series positive ads, promoting his big ideas for the country. We want to know the effect of this decision. The counterfactual will be a world in which Barack Obama decided to instead launch a series of negative ads, attacking John McCain's support for deregulation of the banking and health insurance industries.

We know the Barack Obama's decision will affect the way John McCain responds. If Obama runs the positive ads, then McCain will adopt one strategy, perhaps emphasizing his military service. But if Obama attacks, McCain might retaliate, scolding Obama for dragging the campaign into the mud. In this situation, political scientists refer to McCain's response as a mediating variable. It is caused by Obama's strategy (negative or positive ads) and causes the outcome of interest (respective vote shares).

If McCain responds to Obama's negative attacks, then the ads will have a much smaller effect, perhaps even a backlash effect. On the other hand, if McCain's campaign is the same in both worlds, the negative ads will not be neutralized and the effect will be much larger. The effect of the ads depends on whether the counterfactual scenario allows McCain to respond or holds his campaign constant.

Political scientists allow the other campaign to respond

Typically, political scientists try to figure out how vote shares would be different if one thing changes about the electoral process and the rest of the process was allowed to play out naturally. That is, make one change to the observed process and lets the election naturally run its course. Change Obama's decision to run negative ads, but allow the rest of the process to naturally adjust and allow McCain to respond.

This means that (in addition to studying tiny changes) political scientists allow the candidates to respond. What is the effect of a series of negative ads? Not much, especially since the other candidate will certainly respond with a series of negative ads. Implicit in many claims that campaigns don't matter is the idea that strategic campaigns take steps to minimize campaign effect.

Conclusion: The size of campaign effects depends on whether the other campaign is held constant.

Application for Political Scientists: When claiming that campaigns don't matter much, be clear that you expect the other campaign to respond and have a canceling effect.

Application for Journalists: Understand that political scientists argue for tiny campaign effects because campaigns to cancel each other out.

What does all this mean?

Political scientists think campaigns have small effects, but this claim totally depends on the counterfactuals political scientists use. The counterfactuals used be political scientists and journalists tend to differ in two ways.

  1. Political scientists claim that tiny changes in campaigns have tiny effects. Journals tend to think this means big changes don't matter.
  2. Political scientists know that the campaigns will respond to and counter-act each other's efforts. Journalists often ignore this possibility.
We can confidently say that campaign effects matter a lot or a little depending on the counter-factual used. We can bet that the Romney campaign is a strong force pulling voters in and that the Obama campaign is doing the same. However, campaign variables can be safely left out of predictive models, since...
  1. Campaigns don't vary much.
  2. Campaigns cancel each other out.

...but political scientists should make this clear when claiming that campaigns don't matter much.

P.S. John Sides has a thoughtful response here. You should probably read it.

P.P.S. Jonathan Bernstein also has a thoughtful, though harsher response here. You should probably read it as well.

I encourage you to share this with others and contribute to the conversation at Clarity about the Counterfactual when Discussing "Campaign Effects", which first appeared at more of my thoughts and ideas, subscribe to my blog (via RSS or Email) and follow me on Twitter. You also might like to browse my archive and read my papers on Strategic Mobilization and Testing Hypotheses of No Meaningful Effect.
Follow @carlislerainey

Please comment on the article here: Carlisle Rainey » Methods/Statistics

Tags: , , , , ,