WASHINGTON, D.C. — The question of whether to include paper survey booklets in mail push-to-web designs has become increasingly relevant as survey researchers seek cost-effective data collection methodologies that represent the populations they are studying.
Mail push-to-web designs typically begin with an address-based sample frame, which is a near-perfect list of U.S. households. Households are randomly selected and mailed the survey invitation materials (typically, an invitation mailing and reminder mailings). The mailings traditionally include a link to participate in an online survey, and at least one mailing includes a paper survey booklet that can be mailed back.
As a perceived cost- and time-saving measure, some survey projects may include only the link to the online survey and eliminate the paper survey booklet. Paper surveys can boost response rates and reach populations who may not have the ability or want to respond to an online survey. However, paper surveys introduce printing costs and require longer timelines for design, printing and mailing.
Gallup recently conducted an experiment to explore whether eliminating the paper survey booklet and only offering an option to respond online (web-only) can maintain representativeness without sacrificing response rates, at a similar or lower cost. We sought to address four research questions:
- Does including the paper survey booklet increase response rates?
- When not including a paper survey booklet, can enclosing an additional $1 prepaid incentive in the final mailing help boost response rates?
- Does the inclusion of a paper survey improve demographic representation?
- How do cost and speed compare across designs?
Study Design
Gallup conducted an experiment in October and November 2024, with 12,840 U.S. households randomly selected from an address-based sample frame.1
Households were invited to participate in a survey that covered topics related to politics and economics. The survey fit onto a four-page paper survey booklet (an 11x17 sheet folded into an 8.5x11 booklet). All sampled households were sent three mailings:
- First mailing: A cover letter with the URL and access code for the online survey and a $1 prepaid incentive
- Second mailing: A reminder postcard with the URL and access code for the online survey
- Third mailing: A cover letter with the URL and access code for the online survey
- Half of randomly assigned households received a paper survey booklet and business reply envelope
- Half of randomly assigned households received an additional $1 prepaid incentive
This resulted in four randomly assigned treatment groups:
- Group 1: Survey booklet and additional $1 incentive in the final mailing
- Group 2: Survey booklet and no additional $1 incentive in the final mailing
- Group 3: No survey booklet and additional $1 incentive in the final mailing
- Group 4: No survey booklet and no additional $1 incentive in the final mailing
Results
Response Rates
Including a paper survey booklet and an additional $1 prepaid incentive (Group 1) in the final mailing yielded the highest response rate (13.7%), while the web-only, no-additional-incentive group (Group 4) had the lowest (10.6%).
Although the difference was not statistically significant, the response rate when a survey booklet was included was higher than when it was not included, when comparing equivalent incentives (13.7% vs. 12.3%; 11.8% vs. 10.6%). Including an additional $1 prepaid incentive helped offset what was lost in response rates in the groups not sent a survey booklet (12.3% vs. 11.8%).
Demographics
Including the paper survey booklet seemed to marginally improve the proportion of completes among lower-income respondents, non-Hispanic Black respondents, and respondents with a high school education or less, but these differences were not statistically significant. There were also no statistically significant differences by age, gender, income or political party identification. The paper survey completes accounted for just 10% of completes in the groups that included a paper option. When paper survey completes make up a small percentage of the overall completes, their characteristics must be vastly different from the online completes to move the overall numbers significantly.
The mode of completion revealed significant differences. People who completed by paper survey were older, had lower incomes and lower education levels, and were more likely to identify as Republican than respondents who completed via web.
In other words, receiving a survey booklet did not significantly change the overall demographic composition of our samples, but the people who chose to complete via paper survey — when a paper survey booklet was offered — were demographically different from those who chose to complete it via web. This is not unexpected. Individuals without internet access, and who therefore cannot complete the survey online, tend to be older and have lower income levels.2
Practical Considerations
Cost
The baseline group for calculating cost is the group that did not receive a paper survey booklet or additional $1 prepaid incentive. This group was the least expensive to send, and we will use this as our baseline measure for comparing costs. We calculated a cost per complete — the total cost of all sends (for a given treatment group), divided by the number of completed surveys. The group that included a paper survey booklet but no additional $1 prepaid incentive had the lowest cost per complete ($2.35 below baseline), while the group with no survey booklet and an additional $1 prepaid incentive had the highest cost per complete ($4.36 above baseline).
The groups that included a paper survey booklet had higher response rates, which helped to offset costs. When including a paper survey booklet, fewer survey invitation packets need to be mailed to achieve the same number of completes as only offering an online option. The response rate boost from the additional $1 prepaid incentive, however, was not enough to offset the extra incentive cost.
Field Period
As expected, the survey could be fielded significantly faster when no paper survey booklet was included. Our first web completes were received four days after the initial mailing, and 50% of our web completes were received within the first two weeks. By contrast, the first paper survey complete was received 26 days after the paper survey booklet was mailed, and it took 35 days to reach 50% of our paper survey completes.3
Conclusions
Including a paper survey booklet in a mail push-to-web design improved response rates and was more cost-effective than offering only a web option. Although we found no statistically significant differences in the demographic composition of respondents from each experiment group, including a paper survey booklet did increase the number of completed surveys from several groups that tend to be harder to reach in most survey efforts (low-income adults, low-education adults, non-Hispanic Black adults).
This study was not without limitations. Larger sample sizes may have resulted in greater statistical power to detect differences. This survey also only explored topics related to political and economic attitudes, and results may differ by topic. We also compared a limited set of demographics. There may be important demographic or psychographic differences in those who complete via web versus paper that were not part of our findings. Finally, our findings may have differed had we mailed the paper survey in the first send, followed by additional reminders, which would have provided earlier opportunities for individuals who will not or cannot complete the survey online to respond. We will share the results of a mailing sequence experiment in a future blog.
For studies that require a short timeline, are primarily interested in generating population-level estimates, and do not need to explore subgroups that may have more limited internet access, a web-only option may be an acceptable methodology. However, for studies that can field with a more extended timeline, including the paper survey booklet is more cost-effective and covers individuals who do not have internet access or who may be reluctant to share their responses online.
Stay up to date with the latest insights by following @Gallup on X and on Instagram.
Footnotes
[1] The field period was just before the 2024 presidential election, but we did not observe anything that would suggest the findings were affected by the dates of the field period, such as atypical response rates or daily cumulative number of completes.
[2] Mejía, D. (2024). Computer and internet use in the United States: 2021 (ACS-56). U.S. Census Bureau. https://www2.census.gov/library/publications/2024/demo/acs-56.pdf
[3] It is possible that mail speeds were somewhat delayed by the volume of election-related mailings flowing through the Postal Service, but in our experience, these mailing times are not atypical.
