Polls(ters) Apart

In the wake of the Wisconsin Democratic primary last week, a fight broke out that included charges of incompetence and personal failings, with an underlying intimation that someone was not telling the whole truth. The protagonists, however, were not the presidential candidates, as one might expect, but the pollsters who tell the public what it is thinking. Only this time, according to one pollster, the other pollsters had "goofed" big time about what the public was thinking -- giving a false picture of Massachusetts Sen. John Kerry's lead in Wisconsin and distorting the political landscape in a way that made North Carolina Sen. John Edwards look like a "hero," while setting up Kerry "for a fall."

 

The pollster critic is Warren Mitofsky, a major figure in the polling industry, who currently -- along with his partner, Joe Lenski -- oversees the exit poll operation for the four major networks and The Associated Press. When Mitofsky talks, people listen.

 

On Wednesday, the day after the Wisconsin primary, Mitofsky posted a note on AAPORNET (the Internet list for the American Association for Public Opinion Research, or AAPOR), saying, "Yesterday exposed the biggest polling goofs in my memory." He cited the results of two pre-election polls, one by the American Research Group (ARG) and the other by Zogby International, that showed Kerry's lead over Edwards prior to the primary at 37 points and 27 points, respectively. Kerry won with 40% of the vote to 34% for Edwards.

 

On Thursday, The New York Times reported on the Mitofsky note, and quoted Geoffrey Garin, a Democratic pollster, who said that the news coverage leading up to the primary, based on the ARG and Zogby polls, had set Kerry up for a fall. Mitofsky added that if the polling had been accurate, "nobody would have been surprised that [the election] was close." Instead, the pre-primary polling made Edwards "look like a hero." According to the Times article, John Zogby dismissed Mitofsky's critique of the Wisconsin primary polls as the rantings of "a grumpy old man."

 

This is not the first time that ARG and Zogby have produced controversial polls. Indeed, so frequently are ARG and Zogby polls off from the final election results that ABC News refuses to air either polling organizations' results. The ABC News online politics briefing, The Note, recently highlighted a quote from a Des Moines Register column: "The svengalis at ABC News and some major papers don't like Zogby's tracking. One ABC guru calls this poll ‘crack for the weak.'"

 

The Washington Post reported that Zogby had laughed at the "crack" comment, acknowledging that ABC's polling department had rated his polls as "not airworthy," but he said that his organization was doing "okay without ABC."

What About Late Shifts in Opinion?

Despite the controversies surrounding ARG and Zogby, as well as my great respect for Mitofsky ("grumpy" he is not, "blunt" he is), I think that in this case the charge that these pollsters "goofed" may not necessarily be valid. There appeared to be a late shift in votes toward Edwards after the Sunday night debate, when Edwards criticized Kerry for supporting the NAFTA trade agreement 10 years earlier. All the next day, Edwards hammered on his trade differences with Kerry, suggesting Kerry's NAFTA vote was at least partially responsible for job losses in Wisconsin.

According to the exit poll in Wisconsin, conducted by Edison Media Research and Mitofsky International, 37% of the voters said they had made up their minds either on election day or within the past three days. This group supported Edwards over Kerry by 48% to 25%, suggesting that the debate and subsequent campaigning on Monday had affected the "late deciders."

 

 

 

Given this change in opinion among more than a third of the voters, was it possible that ARG and Zogby were right after all when they conducted their polls several days prior to the election -- and before the Sunday night debate? My analysis suggests the answer could be yes.

The exit poll showed that the "early deciders," voters who had made up their minds in the previous week or earlier, supported Kerry over Edwards by 48% to 28% -- a 20-point difference that is close to what Zogby showed in his last poll from Friday to Sunday (before the debate).

A more detailed statistical analysis estimates that if the "late deciders" had initially (more than three days before the election) supported Kerry with 45% of their vote, and given 32% to Dean and 9% to Edwards, the Zogby poll would have been correct. (ARG's last interviewing finished on Feb. 12, five days before the primary, so it's difficult to perform a similar analysis.)

The statistical analysis cannot prove that ARG and Zogby were accurate in their descriptions of the electorate at the time of their polls, but that possibility cannot be ruled out. It's still possible the pollsters "goofed," but I think it's more likely that large numbers of voters simply changed their minds -- ignoring what the polls told them they were thinking.