A political reporter asks some questions about polling

May 9, 2018
By

(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)

John Kruzel from PolitiFact writes in with some questions:

Readers have asked us to look into Trump’s claim that his support among African Americans doubled as a result of praise from rapper Kanye West. (Trump: “Kanye West must have some power because you probably saw, I doubled my African-American poll numbers. We went from 11 to 22 in one week. Thank you, Kanye, thank you.”)

Trump appears to have been referencing a Reuters weekly poll that showed his approval rating among black men go from 11% for the week ending April 22, to 22% for the week ending April 29.

I’d be grateful if you’d help me explain to readers how reliable an indication this is that Trump is enjoying a favorable shift in public opinion among black men.

(1) The two weekly polls were based surveys, respectively, of 118 and 171 respondents. According to Reuters’ own polling editor, the “credibility interval was more than +/- 9 percentage points for each measurement, which leaves open the possibility that his approval also could have dropped in this time frame.” In layman’s terms, why is having such a small sample size problematic? How might it have affected the reliability of this particular poll?

(2) CNN’s director of polling said the Reuters survey “was conducted using a non-probability online sample, meaning that those who participated signed up to take the poll rather than being randomly selected.” In plain English, what does this mean, and how might it have affected the results?

(3) Given the aforementioned caveats with respect to the poll Trump cited, how accurate is Trump’s statement that his support among African American (men) doubled in a week?

My reply:

(1) The classical margin of error is 2 standard errors, that is, 2*sqrt(p*(1-p)/n). Setting p=0.16 (the midpoint of the two estimates), we get margins of error of 0.07 and 0.06 for the two polls. The classical margin of error for the difference is sqrt(sigma_1^2 + sigma_2^2) = 0.09.

So I guess when the Reuters polling editor said “credibility interval,” he meant 2 times the classical standard error.

(2) I don’t understand the question: the CNN polling director’s statement looks like plain English to me! All I would add is that every opinion poll is a non-probability sample. With nonresponse rates exceeding 90%, it’s doesn’t matter much if the respondents are randomly selected. I mean, sure, why not, but random selection does not give you a probability sample.

(3) I doubt Trump’s support among African American men doubled in a week. Such a statement is indeed consistent with the data, but the data are also consistent with much smaller changes.

P.S. Kruzel’s article is here. It has an error. Kruzel writes:

So while Trump claimed his approval rating among black men for the week ending April 22 was 11 percent, realistically it could have anywhere from 2 percent to 20 percent.

I don’t see how Trump’s approval rate could realistically have been 2%. I mean, sure, 2% is possible—anything’s possible—but given that his support was 11% among the survey respondents, it seems pretty unrealistic to claim that the underlying rate could’ve been 2%.

I guess I’ll rate Kruzel’s post as Half True.

The post A political reporter asks some questions about polling appeared first on Statistical Modeling, Causal Inference, and Social Science.



Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science

Tags:


Subscribe

Email:

  Subscribe