Presidential Primary Polls Conflict: Pick the One You Like?

by D. Cupples |   Over the last couple days, media reporting of a Pew poll's results simultaneously upset some Clinton supporters and excited some Obama supporters. I'm not sure why.

Before significant presidential Democratic primaries this year (e.g., California, Iowa, New Hampshire and Texas), different polls showed different results.  The upshot: some of those polls necessarily turned out wrong -- we just didn't know which ones were wrong until after those elections took place.  Recent  nationwide polls are similarly conflicting.

........................Gallup.......Rasmussen....... NBC/WSJ.........Pew................Fox.........
..................(Mar 24-26)......(Mar. 23-26)......(Mar 24-25).......(Mar. 19-22).....(Mar. 18-19)

Clinton............44%.................46%................45%..................39%...............40%

Obama.............48%................44%................45%..................49%..............38%

Other/Und.........8%.................10%................10%.................12%...............22%

There are a couple of interesting aspects of the five polls in the table above.  Most obviously, the Pew poll's results are vastly different from those of the other four polls.  Either Pew is very right or very wrong.

Another interesting aspect: the percentage of survey participants who are undecided or voting for other candidates is big enough to reverse the results of each of those polls.

That a media outlet or blog momentarily emphasizes one poll or another can give an erroneous impression of how a candidate is really doing in the court of public opinion. 

And let's face it: the media isn't always responsible in its reporting of polling data: often, they just report the data without the explaining the limits or the assumptions on which polling data is based.  I don't know whether that's because of time- or column-inch limits or because some journalists just don't understand basic statistics.

Regardless of which candidate you support, if a single poll's results bum you out, head to Real Clear Politics (a site that publishes tables of polls) and compare multiple polls' results.

That and keep in mind that polls aren't always reliable -- as evinced by the lead up to the New Hampshire primary.  How could they be? 

As mega-pollster John Zogby explained, telephone surveyors go through thousands of people who say "No" (or hang up) before finding as few as 900 people willing to participate in a phone survey. 

Given that a large majority of people contacted would prefer to NOT take phone surveys, it seems likely that the polls don't actually reflect what the majority of voters are thinking.

Including the few posts linked below, BN-Politics has more than 30 posts (starting back in May 2007) dealing with polling data or the media's handling of it.

Other BN-Politics Posts:

* Hillary's Health Care Plan Includes Caps (and Ways to Help Pay for it)

* Questionable Pre-Primary Polls re: Ohio and Texas

* Primary 2008: Polls Don't Seem to Mean Much

* It's not Just NH: some Polls Favored Wrong Candidates Before Iowa

* Polling Data Inadequately Reported

* Hoping for More Careful Analysis of Polling Data

.

Comments

[Adam]

Someone needs to do a really really big meta-study of all these polls, and figure out if there are trends in the selection bias between people who are willing to take the surveys and those who hang up.

It sure seems like the Rasmussen poll has 10% undecided, even if they failed to report it explicitly.

[D. Cupples]

Adam,

I'm soooo embarrassed. My mind played tricks: I added 44 and 46 and came up with 100 (though I really do know that it = 90).

Thanks for pointing out the error so I could correct it quickly.

I'm not sure HOW they could figure out sampling biases given that they need input from the very people who refuse to participate.

It's been years since I took stats. I'd have to look up how to calculate a z-score at this point.

I do remember thinking (while in class) that some questionable assumptions were involved in the process of calculating error margins. This troubled me.

I have no idea why the polls were generally right about SC and the Potomac states but conflicted re: the states I list in this post.

Personally, I wouldn't join a big-stakes betting pool based on political polling.

[Adam]

My guess on the resons for the bad polling in those four states:

California and Iowa: poor assumptions on who was a "likely voter"

New Hampshire: big news story right on the eve of the election threw things off.

Texas: last minute ads had a significant impact, plus the "Rush effect".

Off the top of my head, a meta-study would look at things like the demographics of those who answer the surveys compared to those who do not. You then match this up against the demographics/results of exit polls, and finally the actual results. If you have enough data, you can probably tease out significant demographic and voting trends in those who answer surveys versus those who don't. You may figure out that, say, older voters who support Obama are more likely to answer the survey than older voters who do not, but no such trend exists among middle-aged voters. Stuff like that.

The thing about any sort of betting pool is, you don't need to be right all the time, you only need to be right more than average. I can do well at things like CNN political market just by taking the obvious bets.

[D. Cupples]

Adam,

WHAT assumptions about likely voters? The surveyors ask people whether they're likely voters, no?

As for the other states, did you even look at the tables that I'd linked to in the post? NOT all polls were wrong about CA, TX, OH.... Only some were.

That was one of my points: when polls of a similar time conflict, some of them are necessarily wrong. That's because of inherent problems with polling (or sampling) as a whole.

Unfortunately, I don't know precisely what those inherent problems are.

Tangent: another fundamental problem with phone surveys is that they rely on self-reporting. Some people say things that aren't true because they don't want to look stupid or bad to the surveyor. Some just lie for the hell of it. Nobody knows the percentages.

I realize that self-reporting is the best we have. At the same time, I'm ever aware of some of their inherent flaws of polling generally.

[Adam]

As I understand it, some polls do just ask whether you plan to vote, but others ask correlative questions (e.g. "did you vote in the last primary?") and throw out or discount answers from people they consider unlikely to vote. Different polls use different recipies in their secret sauces, hence varying results.

So, for example, I'd guess that there's something about Zogby's determination of a likely primary voter that has tended to favor Obama. Hence the wacky Texas/Ohio numbers. Everybody else was within margin of error, I think.

You're absolutely right that there are other biases in polling. There are some famous cases of this. For instance, polls, including exit polls, showed David Duke (the former KKK wizard) getting blown out in the Louisiana governor's election, but he ended up only losing by a very narrow margin. People were embarrassed to admit they supported Duke.

[Adam]

I just found another poll-aggregating website that is focussing on the general election. This site has a good handle on basic statistical analysis, which makes for some interesting insights.

http://www.fivethirtyeight.com/

[D. Cupples]

GREAT site, Adam. Thanks.