Friday, January 21, 2005

Is this the whole story?

Yahoo! News - Firms report flaws that threw off exit polls Somehow, I'd be more likely to beleive an explanation more like the one given for why much of the data from the Doppler Wind Experiment was lost.

7 Comments:

Blogger Justin said...

First, they doppler wind experiment failure made me heartsick... 18 years lost because someone forgot to turn it on. I do hope that someone will be able to recover some of the data from the deep space radio network, though.

On the exit polls, I can't buy the explanations-- there isn't enough variation in the underlying data sets to explain the overall variation.

Men vs Women
From http://www.cnn.com/ELECTION/2004/pages/results/states/US/P/00/epolls.0.html

Men were 46% of voters and went 55/44 for Bush
Women were 54% of the voters and went 51/48 for Kerry

Let's assume that we horribly mis-sampled voters by gender so that we were getting only half as many men (23% of the sample) and the other 77% were women. In that scenario, Bush still wins, but just barely (49.61% to 49.39%). The polls would have overstated the margin by 3.22%, half the actual spread. I have to believe that any idiot would figure out immediately that more than 1 of every 4 voters was male and adjusted appropriately.

Young vs. Old-

Again, let's assume that we were oversampling 18-29 year olds and undersampling the over 60 crowd. Adjust the 18-29 vote from 17% to 25% of the total and adjust the over 60's from 24% to 16%.

Bush still wins 50.56% to 48.6%, but the margin of victory shrinks by 1.96%

Denied access to polling places

Don't see how this is a factor. It may have reduced their sample size, but it shouldn't have skewed the results.

Even adding the effects together, we don't get to a margin difference of 6.5%, and we're assuming some hugely significant skews in data. I have a hard time believing that pollsters, even insufficiently trained ones, would not notice a glaring absence of men or that they weren't getting anywhere near enough senior citizens. I suppose the pollsters could have been talking all day to women aged 18-29, completely ignoring men over 60, but that seems doubly unlikely.

1/21/2005 02:17:00 PM  
Blogger Eric said...

I agree with your analysis. I suspect there might be more to the real explanation. Perhaps some higher ups think this sounds easier to fix for next time than admitting that their analysis team didn't know how to analyze the data properly.

I might beleive that if the 18-29 year old males tended to interview 18-29 year old females. But unless they have very skewed hiring practice, I'd expect about 50% of the interviewers to be female, so 75%/25% women/men seems like the absolute max. And I do has some small bit of faith in humnaity that tells me at least some 18-29 year old males did take this more seriously than a chance to jump start their love life.

Some degree of senior citizen bias might be somewhat plaussible. However, I agree it's not enough to explain their mistakes. And since they asked the age of the interviewee, and they presumablely knew the age of the intervier, they should be able to measure and correct for any biases there. If they were an experienced polling organization, then I would expect them to have a prior estimate to use in the early interpretation before the new information from this years polls could be incorporated to refine the correction.

1/21/2005 02:46:00 PM  
Blogger Eric said...

Thanks for the quantitative analysis. Now you've justified my suspicion that there's more to the story than their admitting. Am I the only one who feels that our generation takes a disporportionate ammount of blame (in general, not just here)?

I have to wonder if they think that it's easier to convince people to trust them next time, if they say their errors in 2004 were due to mistakes by young poll takers (who will probably be mostly different each election anyway), rather than mistakes by the people who analyze the data.

1/22/2005 09:17:00 PM  
Blogger Justin said...

It's bad enough that the polling organization is floating an explanation that is, at the least, incomplete. What really riles me, though, is that the folks that are reporting this don't bother spending 60 seconds to do the analysis and conclude that there is something rotten in the state of Denmark.

1/22/2005 11:54:00 PM  
Blogger Vincent said...

The Mystery Pollster has consistently produced a good analysis of what may have gone wrong with the exit polls, as well as debunking myths about other possible causes. It merits sorting through the archives a bit.

1/23/2005 12:00:00 AM  
Blogger Eric said...

Their latest post doesn't appear to take a very critical stance towards the internal report.

1/23/2005 06:40:00 AM  
Blogger acg said...

Well, after I voted, the one poller I saw was, in fact, hitting on another female voter. She didn't want to tell him how old she was, either.

1/25/2005 05:03:00 PM  

Post a Comment

<< Home