Polling, Likely Voters, and the Law of the Commons

As our tradition has been in presidential election years, Gallup's focus this year was on producing an estimate of the national popular vote. We don't "predict" the election, nor do we make estimates of the Electoral College. In the end, Gallup's national popular vote estimate was that the popular vote was too close to call, a statistical tie -- 50% for Mitt Romney, 49% for Barack Obama. When the dust settled, Romney got 48% of the popular vote and Obama received 50%, meaning that Gallup's percentage-point estimate was within two percentage points for Romney and within one point for Obama. The "gap" difference was three points. All of these are well within the statistical margin of error and underscore the accuracy of random sampling today, even with all of the challenges provided by changing forms of communication (i.e., cellphones), changing demographics, lowered response rates, identifying likely voters, and a wide variety of other factors.

We would always like to make a final poll estimate that is exactly on target with the final popular vote percentage for both candidates. That is the goal. But our estimate (and almost all other national polls at the end), gave a broadly accurate picture of what was, in fact, a very close popular vote.

As we have always done, we here at Gallup will continue to review all aspects of our election polling, from sample development, to interviewing, to data processing, to statistical weighting, to final analysis. Already this year, our methodologists instituted a number of changes. As explained here, we increased our percentage of interviews conducted on cellphones to 50%, modified our weighting procedures, tweaked other aspects of our sampling, and changed some aspects of our likely voter procedures.

We will continue to examine our likely voter procedures, a real key to understanding the final popular vote. Our final estimate of registered voters was an unallocated 49% for Obama, 46% for Romney. The transition to likely voters moved that to the unallocated 49% Romney, 48% Obama.

We have modified our likely voter procedures in a number of ways over the years since they were first developed by George Gallup and Paul Perry decades ago. But I think it is clear that voting today is subject to new pushes and pulls, including, in particular, the highly sophisticated ground games employed by the Obama (and, to a lesser degree, the Romney) campaign this year. These methods may in the end affect voters who were not certain about voting at the time of a poll interview, but who were brought into the voting pool at the last minute by aggressive get-out-the-vote and late registration methods. Our traditional "bootstrap" method of identifying likely voters is self-weighting -- letting voters' responses to questions determine their probability of voting. This bears investigation. We will use the government's post-election data, along with internal evidence, to see if further assumptions, investigations, or changes might be necessary.

We do believe that the presidential campaign underwent significant changes as it progressed this year. Romney clearly gained as a result of the first debate in Denver, and he held onto at least a marginal lead position in our polling until the week before the election, when Superstorm Sandy hit. Obama gained five points on the gap between our last pre-storm polling and the final poll. It may be that he continued to gain on into Election Day.

Gallup uses a wide variety of CPS demographic weighting targets for all of our samples. We traditionally do not put governors or other controls on party identification, as someother polls do. We reported results using very large samples this year, which generally helps control for normal sampling variation, but we will continue to look at this process.

Changes are in the wind that may affect polling as we know it in the years ahead. I think it is likely that we could see significantly fewer polls conducted in the 2016 election -- at the state and national level -- than in this election. In fact, some reports already say there were fewer state polls conducted in this election than in 2008, so we may already be seeing the beginning of a trend.

Some of this is a result of budget cutbacks, and some will be a shift to the use of other technologies for assessing public opinion in the future.

But some of this will result from a variant of the venerable "law of the commons." Individual farmers can each make a perfectly rational decision to graze their cows on the town commons. But all of these rational decisions together mean that the commons becomes overgrazed and, in the end, there is no grass left for any cow to graze. Many individual rational decisions can end up in a collective mess.

We have a reverse law of the commons with polls. It's not easy nor cheap to conduct traditional random sample polls. It's much easier, cheaper, and mostly less risky to focus on aggregating and analyzing others' polls. Organizations that traditionally go to the expense and effort to conduct individual polls could, in theory, decide to put their efforts into aggregation and statistical analyses of other people's polls in the next election cycle and cut out their own polling. If many organizations make this seemingly rational decision, we could quickly be in a situation in which there are fewer and fewer polls left to aggregate and put into statistical models. Many individual rational decisions could result in a loss for the collective interest of those interested in public opinion.

This will develop into a significant issue for the industry going forward.

The U.S. government, of course, does not conduct political polls. Perhaps it will be necessary to develop a consortium of pre-election polls in the future, like the National Exit Poll -- one gigantic poll or set of polls for all interested organizations. Or, on the other hand, individual polling organizations may come together in the future and sponsor one consortium of analysts making sense of the individual polls.

But clearly the traditional process, by which individual organizations independently make decisions to conduct their own polls, may be challenged in coming elections.