Statistical Noise in Election Polls: Obama and Black Vote Poll - Inaccurate, Hype

From [HERE] Some headlines on Tuesday suggested that Barack Obama has a problem among his African-American base. The warrant for this claim is a new Public Policy Polling survey in North Carolina, in which Mr. Obama won only 76 percent of the black vote, compared with 20 percent for Mitt Romney. Mr. Obama won 95 percent of the African-American vote nationwide in 2008, and 95 percent in North Carolina, according to exit polls.

This is not a great poll for Mr. Obama. Public Policy Polling, which conducts polls for Democratic clients as well as issues them on its own behalf, has had a Democratic “lean” so far this cycle. Mr. Obama trails Mr. Romney by 2 points in their North Carolina poll, and that might imply a somewhat larger disadvantage for him if another polling firm were to survey the state.

Our forecasting model had been pessimistic about Mr. Obama’s chances to win North Carolina even before this poll was released, giving him just a 30 percent chance of doing so.

But if the overall numbers of the poll are of some significance, the notion that Mr. Obama is in much danger of losing his African-American voters is not really justified by the survey. The reason is simply that there isn’t a sufficient sample size to come to much of any determination about the question.

The poll surveyed 810 North Carolina voters, and weighted its demographics such that 22 percent were African-American. If you do the math, that suggests that about 180 respondents in the survey were black.

The margin of error on a 180-person subsample is quite high: about 7.3 percent.

And that probably underestimates the real margin of error somewhat. The 22 percent African-American figure in the poll represents the results after demographic weighting. However, some demographic groups are considerably harder to get on the phone than others, African-Americans among them. Since the incidence of cellphone use is considerably higher among African-Americans than among white Americans, this difficulty will be especially true for a polling firm that does not include cellphone voters in its sample, as Public Policy Polling’s surveys do not.

In practice, it is possible that Public Policy Polling got something like 100 African-American voters on the phone rather than 180, which would be associated with a margin of error of about 10 percent. In addition, all of these African-Americans were reached on landlines, and they may have different demographic characteristics (for instance, being older) than the cellphone voters that the survey missed.

A more robust sample of African-American voters comes from the Gallup poll, which interviews about 3,500 people a week for its national tracking poll and does include cellphones in its sample.

The Gallup poll suggests that Mr. Obama’s ratings among African-American voters may have dropped by a percentage point or two over the past few weeks, but probably not more than that. In the four full weeks of polling since Mr. Obama announced his support for same-sex marriage — which had been unpopular in the African-American community until recently — his approval rating has averaged 86 percent among black voters. That compares with 87 percent for 2012 as a whole.

Mr. Obama’s approval rating among black voters in Gallup’s most recent week of polling, which ran from June 4-10, was 85 percent.

There just isn’t any indication that there is much going on here, other than statistical noise. Although Mr. Obama may not win 95 percent of African-American voters in 2012, he will surely win an overwhelming majority of their support. Mr. Obama has much bigger problems to worry about.

There is nothing theoretically wrong with looking at Mr. Obama’s support among different demographic subgroups. But doing so can lead to some bad analysis — sorry, there’s no polite way to put it — if you aren’t alert to this very basic issue of sample size.

Furthermore, keep in mind that in addition to the already-high margin of error related to polling demographic subgroups, the margin of error is calculated in such a way that it is supposed to cover 95 percent of all possible cases. But it won’t cover 100 percent. The results from 1 in 20 polls — or 1 in 20 subsamples within polls — will fall outside the margin of error.

There are roughly six presidential polls being released on an average day right now, and the number is sure to proliferate in the coming weeks. Each of those six polls might include demographic breakdowns for, say, 15 demographic groups.

That means that on a typical day, you might have 90 demographic subsamples to pick from. Several of these results almost certainly will fall outside of the margin of error — they will be outliers.

Unfortunately, the news media has a tendency to focus on the outlying results, while missing the consensus trend. The most implausible results often get the most attention.

Meanwhile, we have entered an era where different polling firms — including Public Policy Polling — have substantial “house effects” or partisan leans. Margin-of-error calculations assume that pollsters are doing everything right — other than taking a random sample of a population, thereby introducing sampling error. But that is too forgiving a view. Polling firms introduce lots of “unforced” error into their polls because of the methodological shortcuts, like the failure to call cellphones.

We’re now at a phase of the election cycle where there is just enough information to get yourself in trouble. With a half-dozen polls being released every day, you’ll have some ability to shop around in support of whatever narrative you might wish to advance. But there is not yet so much data that all these anomalies will wash out in the average.

An appreciation for the noise intrinsic to polling — both that which is inevitable because of sampling error, and that which is introduced by polling firms because of improper methodology — is crucial if you want to have a well-informed idea of what will happen in the election.

Demographic subgroups are associated with much higher margins of error than the overall numbers in a poll. So, weaning yourself from overreacting to every shift in the demographic numbers is a good first step to avoid being driven crazy between now and November.