Jeff Jones, Frank Newport, Lydia Saad
We have received several questions asking why our two Monday releases on national Democratic nomination preferences differ.
To review: The Feb. 21-24 USA Today/Gallup poll shows that 51% of national Democrats and Democratic-leaning independents prefer Barack Obama and 39% prefer Hillary Clinton. The Feb. 22-24 Gallup Poll Daily tracking results showed a much tighter race, with Obama at 47% and Clinton at 45% among a national sample of Democratic and Democratic-leaning voters.
On Monday, two other national polls released divergent results. The New York Times/CBS poll showed Obama with a significant 16-percentage point lead over Clinton. The Associated Press/Ipsos poll showed Obama with a three-point lead. These two polls were conducted roughly contemporaneously with Gallup's polls, and may help underscore the finding that there is a great deal of fluidity within the Democratic electorate at this point, who are choosing between two candidates they like very much and have similar platforms.
In some sense, a focus on the gap between candidates exaggerates the differences between polls. Polls that show Obama with a 12-point lead versus a two-point lead may be more striking than the fact that Obama has four points more support in one poll than the other and Clinton has six points less in one poll than the other.
Given the importance attached to Gallup polls' numbers, it is important for us to try to gain some understanding of why Gallup Poll Daily tracking and USA Today/Gallup poll differ, and share that information with the interested community.
Gallup probably does more national polling than any other firm does, so this is certainly not the first time and won't be the last time that one of our polls produces results inconsistent with other polls. When this does happen, though, we do our best to analyze why those differences might have occurred. Unfortunately, these investigations rarely yield a proverbial smoking gun, and, beyond normal sampling error, it is not clear why one poll might differ from other polls -- even in cases when the differences between polls appear to be beyond the margin of error. This appears to be the case with Gallup Poll Daily tracking and USA Today/Gallup poll as well.
In general, the methodology between the two polls is similar. Both rely on the same RDD samples and respondent selection procedures (adult with most recent birthday), both attempt to interview the proper proportion of men and women by region per 1,000 interviews, and both final samples are weighted to national demographic norms, using similar weighting approaches.
There are some minor differences between the two procedures Gallup uses to conduct its tracking polls versus its non-tracking polls, but these differences do not explain why Obama has a much larger lead in one poll than the other poll.
Dates of interview: The USA Today/Gallup poll started on Feb. 21, a day Gallup did not conduct Gallup Poll Daily tracking interviewing. But an analysis of the night-by-night numbers indicate that Feb. 21 was not a particularly strong night for Obama. His best days in the USA Today/Gallup poll were Friday, Saturday, and Sunday, Feb. 22-24, and the numbers were consistent on each of these days.
Differences in sample: The USA Today/Gallup is reported on the basis of all Democrats and Democratic-leaning independents. The Gallup Poll Daily tracking is reported on the basis of Democrats and Democratic-leaning voters -- those who say they are "extremely likely," "very likely," or "somewhat likely" to vote in their state's primary or caucus or report that they have already voted. To provide an apples-to-apples comparison, Gallup Poll Daily tracking would have shown a result of 46% for Obama, 45% for Clinton among all Democrats and Democratic-leaning voters.
Interviews by night: Gallup interviews an equal number of respondents per night (1,000 per night, which yields more than 400 Democratic "voters" each night) for Gallup Poll Daily tracking. Gallup non-tracking polls interview declining number of respondents per day, with the most interviews conducted on the first and second nights, and the final one to two nights largely consisting of callbacks of previously dialed numbers. But because the nightly numbers did not differ much in the USA Today/Gallup poll or in Gallup Poll Daily tracking, this is not an explanation.
Minor interviewing differences: Gallup conduct interviews in Spanish when necessary for Gallup Poll Daily tracking, but does not normally conduct interviews in Spanish for the USA Today/Gallup poll. Both polls include interviews with cell phone-only respondents, and although there are differences in the cell phone procedures, we weight both samples in a similar fashion. Both samples had about the same proportion of cell phone-only interviews, and the results of candidate preference by phone type did not differ enough to explain the dissimilarity in results.
Question order: We often don't know how the order of questions on a survey will affect the respondents' answers. In Gallup Poll Daily tracking survey and USA Today/Gallup poll, we ask the ballot questions toward the beginning of the questionnaire. Specifically, the Gallup Poll Daily tracking survey asks the ballot question immediately after the respondent selection and party ID screening questions. The USA Today/Gallup poll asked the ballot questions following the respondent selection questions, likely voter questions, presidential approval, candidate favorables, and party ID screening questions. To date, we have not observed much variation in the USA Today/Gallup trend when we ask the ballot question earlier or later in the survey, before or after favorables, etc. However, this does not rule out the possibility that the order of questions on the survey did have an impact on the results of our recent polls.
The demographic compositions of the two samples for Gallup Poll Daily tracking and the USA Today/Gallup poll were similar, and the relative proportions of subgroup members supporting Clinton or Obama vary by acceptable margins.
Download the ZIP (4 KB) file to view the Gallup polls compared.
To reiterate, none of the known differences between Gallup Poll Daily tracking and the USA Today/Gallup poll, based on our analysis, are obvious causes for the disparity in the Democratic ballot estimate between the two samples. In fact, the estimates of the Republican horse race between the two samples were almost identical. Previously, when we conducted Gallup Poll Daily tracking and the USA Today/Gallup poll simultaneously, the Democratic estimates were within the margin of error of each other.