With nine in 10 adults owning a cellphone and 72% of smartphone owners reporting that they check their phone at least once an hour, text messaging (SMS) has become the communication norm for much of the U.S. population. Given its popularity, text messaging appears to be a useful way to contact survey respondents, particularly those who tend to have lower response rates with traditional survey methods, such as young adults.
However, Gallup's research and experiments suggest that text messaging is not yet ready for prime time in survey research. This mode of contact has legal limitations, yields lower response rates than telephone surveys and restricts question length. While text messaging could be a supplementary way to reach respondents, Gallup does not recommend using it as the only means of contacting the general population.
How Text Message Surveys Work
Text message surveys can be administered in two different ways. The first option is to text questions and answers back and forth, which is advantageous because anyone with a cellphone can respond. But this method also has significant limitations. Questions can be no longer than 160 characters, including the question wording and response options. Messages that are longer than 160 characters are broken into segments by the device before it sends, and the recipient's device rebuilds the message. While some devices rebuild the messages so that they appear as one cohesive message, messages may not be received in the correct order.
The number of questions must also be kept to a minimum. Questions and responses are sent one at a time, and our research finds that respondents tend to lose interest more quickly than with other modes of data collection.
The second option is to text respondents a link to a web survey they can complete from the browser on their phone. Although this option gives researchers greater flexibility with question wording, about 25% of the population does not own a smartphone and will be unable to launch a web survey, and those who do have a smartphone may have to use or pay for data to complete the survey.
It is important to note that FCC regulations make it illegal for companies to send text messages without expressed consent. This means Gallup or its clients must have explicit consent from respondents before sending them a text message survey. Simply having permission to contact the respondent via cellphone is not enough. Individuals must give consent to be contacted via text message, which is a major obstacle for most survey projects.
Experiments Reveal the Pros and Cons of Text Message Surveys
Because consent to contact via text message is required to send a text message survey, Gallup used its Daily tracking survey and the Gallup Panel as the sample sources for two text message experiments. In both samples, Gallup asked respondents if it could occasionally send them surveys via text message. The experiments were conducted among those who agreed.
Experiment 1
In the first experiment, we wanted to see how an SMS survey compares with a telephone survey in terms of response rates and substantive responses. Gallup randomly assigned respondents who had provided text message consent to one of three treatment groups: a traditional telephone survey administered by a live interviewer, a text message survey or a text message with a link to a web survey. Two questionnaire lengths were also tested: five questions and 12 questions.
Gallup asked all treatment groups about their satisfaction with the ways they can access banking services. All treatment groups received the same question topics and response options. However, the text message questions were shortened because of the 160-character limit.
Example Wording
Phone and Web
Please rate your overall satisfaction with each of the following ways of interacting with your bank using a five-point scale, where 5 means you are extremely satisfied and 1 means you are not at all satisfied … Your most recent in-person visit to a bank branch to talk with a teller or someone else?
SMS
From 1 (not at all satisfied) to 5 (extremely satisfied), how satisfied are you with your most recent bank visit? Use 6 for "Don't know/use."
Results
Response rates for the SMS-to-web surveys (12% for five questions and 11% for 12) and SMS-only surveys (12% for five questions and 13% for 12) were significantly lower than response rates for phone surveys (38% and 41%). Breakoffs, the number of people who start but do not complete the entire survey, were low in all treatments but were slightly more likely in the SMS-only 12-question group, at 19%.
| Questionnaire length | Response rate | Breakoffs | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| # of questions | % | % | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Survey mode | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| SMS | 5 | 12 | 8 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| SMS | 12 | 13 | 19 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| SMS-web | 5 | 12 | 1 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| SMS-web | 12 | 11 | 10 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Phone | 5 | 38 | 0 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Phone | 12 | 41 | 0 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Gallup | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Experiment 2
In the second experiment, Gallup tested sending survey invites and reminders to Gallup Panel members via email and text message. Respondents were randomly assigned to one of three treatment groups: email and text invites and reminders, email invites and reminders only, and text invites and reminders only. All emails and text messages directed respondents to a web survey. For analysis, we split respondents into four propensity groups based on the number of surveys they had responded to in the past.
Results
For the lowest-propensity responders, the mode of contact made no difference. For all other responders who had participated in at least 25% of previous surveys, response rates were highest when using a combination of email and text reminders. This finding is consistent with other research, which has found that employing a variety of contact methods can increase the likelihood of participation.
| SMS only | Email only | SMS and email | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| % | % | % | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Respond to 1% to 24% of invitations | 2 | 1 | 3 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Respond to 25% to 49% of invitations | 25 | 22 | 39 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Respond to 50% to 74% of invitations | 47 | 63 | 78 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Respond to 75% to 100% of invitations | 77 | 94 | 96 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| *Respondents were grouped into one of four response propensity groups based on the percentage of survey invitations they had responded to in the past. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Gallup | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Text Message Surveys Do Not Improve Sample Composition
In experiment one, Gallup found no demographic differences between the SMS, SMS-to-web and phone responders in terms of gender, race and income level. However, individuals who responded to the survey via text message were on average slightly younger (43 years old versus 46 years old) and tended to be more highly educated than those who responded via telephone or the text to web.
In experiment two, Gallup found that the use of text messages in combination with email to send invitations and reminders increased participation across all demographic groups, including age and education level. The exception was among Hispanics, who had no significant response rate difference between those who were contacted via email only and those who were contacted via email and text.
Based on the results of these two experiments, Gallup sees no evidence that text messaging will significantly improve the composition of survey samples.
Conclusions
Currently, the major obstacle for conducting a survey via text message is obtaining the consent to send a message. This legal barrier greatly limits the scope for conducting SMS surveys.
Organizations that do have permission to conduct text message surveys should consider the limitations of this mode of contact. Gallup discovered that SMS surveys have lower response rates than traditional phone surveys and present challenges concerning question length. The 160-character constraints on SMS can lead to surveys with less-than-ideal wording and can disrupt trending if other modes of contact were used in the past.
Based on the results of the experiments and the limitations of SMS, Gallup does not recommend conducting stand-alone text message surveys in the U.S. for obtaining representative estimates. However, text message surveys have promise as part of a mixed-mode design and can significantly improve response rates when used in combination with email communication. There may be other potential ways to use SMS in a mixed-mode design, such as sending advanced notice for a telephone survey. Gallup will continue to explore the value of using text messages in survey research and will share findings in future blog posts.
Read more from Gallup's Methodology Blog and visit our Methodology Center.
