George Gallup and the pioneers of public opinion polling were committed to accuracy, neutrality, objectivity and transparency in publicly released surveys. Gallup always strives to uphold those ideals, and our associates fully commit themselves to fulfill George Gallup's mission and deliver the most accurate and objective research for our clients and partners and share it with the public. This openness gives academics and researchers the ability to review our findings with sufficient data to inquire about methods, weightings, sample frames and findings. It is crucial to the integrity of social science findings. On occasion, this transparency leads to the discovery of missteps or oversights.
In this release, we detail the methodological research principles used in our U.S. public opinion research and describe an error we made on the 2020 "American Views: Trust, Media and Democracy" report sponsored by Knight Foundation. This error did not affect any of the major conclusions in the report as the revisions to the numbers that resulted were mostly slight.
Gallup attempts to fulfill its mission by using the highest quality methodology when conducting surveys. This includes writing survey questions to avoid known biases in how people answer questions, relying on an inclusive sample frame of the population being studied, random sampling within that frame to ensure representative samples and using statistical procedures to address survey nonresponse.
High-quality surveys begin by selecting a random sample of the population to be studied, and in the case of the Knight Foundation study, it was U.S. adults aged 18 and older. Random sampling ensures that every member of the population has a known probability of being selected for the survey, a key pillar of statistics and survey methodology.
The sample frame -- the list from which respondents are randomly selected -- ideally should cover 100% of non-institutionalized adults aged 18 and older. The frame typically contains contact information, such as phone numbers, addresses or emails that determine the mode researchers can use to contact respondents. In the "American Views" study on behalf of Knight Foundation, Gallup used an address-based sample (ABS). The ABS sample frame is created using the Delivery Sequence File of the United States Postal Service. ABS sample provides near-perfect coverage of all U.S. households and is considered a gold-standard methodology that is also used in many large-scale government studies. Besides providing excellent coverage, ABS has other benefits. Research, including Gallup's own testing, has found that ABS studies can achieve significantly higher response rates than Random Digit Dial telephone surveys. Further, ABS allows researchers to know with certainty respondents' geographical location. ABS also gives researchers the flexibility to contact households via mail and ask them to complete the survey via mail or web, though the Knight Foundation survey was conducted entirely by mail.
In an ideal world, every person sampled for a given survey would take it, and the sample would be nationally representative with little to no adjustments necessary. However, many people selected for surveys do not take them. This is called nonresponse, and it occurs in all types of surveys, including those conducted by mail, web and telephone. Furthermore, survey participation and nonresponse rates often vary in consistent ways by subgroup. Certain subgroups -- racial and ethnic minorities, young adults, people with low income and less education -- tend to participate in surveys at lower rates than other groups. That means almost every sample a survey researcher obtains will have proportionately fewer young adults, fewer racial and ethnic minorities, fewer people with low income and fewer people with lower educational attainment than the population overall. Consequently, samples tend to have disproportionately more older adults, white adults, middle- and upper-income earners and college graduates.
To correct for this nonresponse, survey researchers adjust or "weight" their sample so that it matches U.S. population characteristics. For example, young, minority and lower-income respondents will be given more weight in the weighted sample, and older, White and middle- and upper-income respondents will be given less weight in the weighted sample. After these weights are applied, the weighted sample represents those groups in their correct proportions -- or very close to it -- in the U.S. population.
Gallup recently learned of possible anomalies in weighting by age and education in the Knight Foundation-sponsored "American Views" study, a mail survey of more than 20,000 U.S. adults conducted in late 2019 and early 2020. Upon learning of this possibility, Gallup researchers responded immediately and reexamined the data. During this review, they found that the weighting by age was done appropriately and accurately; however, they confirmed an issue in weighting by education. They then conducted a root cause analysis to understand the source and implications of the error in weighting by education. When calculating weights, "targets" for the U.S. population are obtained from gold-standard sources such as the U.S. census. During the weighting of the "American Views" survey, the education categories used in the survey were incorrectly mapped to the categories used in the census data that were providing the weighting targets. The result was that responses from college-educated respondents received too much weight and were overrepresented in the weighted sample. A second Gallup methodologist reviewed the corrected weights and confirmed the education weighting was done accurately. As part of a larger review of the weighting, this methodologist also proposed a possible adjustment in the weights used to account for differences in respondent selection within household for some respondents. The modified respondent selection adjustment, when applied, only caused a few isolated percentages to round differently, so it was not included in the final computed weights.
Gallup notified Knight Foundation of the issue on Oct. 26. Knight Foundation posted a message to the report release landing webpage notifying people of the error, informing them the revised report would be posted on Nov. 9 and giving them a means of contacting Gallup about report questions in the meantime.
Gallup researchers re-ran the data with the corrected weights and reviewed all materials produced based on the results to identify discrepancies between the content and the new results. Since many of the substantive questions in the survey did not show meaningful differences between college graduates and non-graduates, the weighting error did not raise obvious red flags that pointed to a larger data quality issue.
Upon updating the executive summary of the report, none of the eight major conclusions needed to be revised. Some references to changes from the 2017 results have been removed because the differences are no longer meaningful, but the overarching conclusions about public opinion of the media landscape and the polarization of those views in 2020 remain unchanged. Gallup also reexamined all the recent studies it has conducted for Knight Foundation, including the 2018 "American Views" study, and found the weighting was correctly applied in those studies.
Gallup editors and graphic designers thoroughly updated all the materials associated with the report, and Gallup researchers created supporting documentation about the error. The corrected report and all supporting materials, including the dataset, were made available on the Knight Foundation website on Nov. 9.
These corrective measures, including this release, are part of Gallup's commitment to accuracy and transparency. All Gallup reports undergo a robust quality assurance and editing process, including being reviewed by the lead researcher, multiple peer reviewers, and a content and copy editor. That process did not identify the weighting anomaly in this instance, so Gallup is reviewing all aspects of the existing process. Gallup is taking key process improvement steps to ensure a weighting error of this nature does not happen again. These steps include:
- All analysts and quality assurance partners on public release projects will reaggregate demographic variables from the raw, uncoded data to ensure target and observed data categories are aligned.
- Gallup will create a clear set of decision criteria for when to apply different levels and tools for quality assurance for weighting to different types of research projects. A project such as the "American Views 2020" study will receive the full suite of quality assurance steps, including weighting checks by a second independent statistician.
- Gallup enlisted the help of a group of outside research advisers, including Dr. Michael Traugott and Dr. James Lepkowski from the University of Michigan Survey Research Center, to independently confirm that the new 2020 study weights are accurate given the assumptions made to create the weighting plan and targets for this study. These advisers will also make recommendations on how Gallup can improve its quality assurance process for weighting and suggest ways in which Gallup might improve its overall weighting approach going forward.
The Gallup team that supports Knight Foundation will follow these process steps and will conduct regular protocol audits to ensure that all enhanced weighting processes are followed.
At Gallup, we are highly dedicated to contributing to public discourse about the issues that matter most to the future of the U.S. through the highest quality research. While in this case, we made a rare and unfortunate error, we are committed to full transparency, independent review of our work and process enhancements to mitigate any risk of a similar error in the future. Although many figures in the report changed after the corrected weighting, the main conclusions of the report remain unchanged, and we believe the report makes a substantial contribution to our understanding of trust in media.
At Gallup, we believe that complete openness and a commitment to accuracy is an integral part of informing the public and ensuring that researchers follow the best possible processes and reporting guidelines. Gallup will continue to show its commitment to these standards and correct the record when necessary.
If you have any questions about the data, please email usviews2020@gallup.com.
Jeffrey Jones, Ph.D., Senior Research Consultant
Jenny Marlar, Ph.D., Research Director, U.S. Social Research
Ilana Ron Levey, Senior Director, Public Sector Consulting
