Probing is a common technique that researchers use in interviewer-administered surveys when respondents initially refuse to answer a question or say they "don't know." Interviewers are trained to use neutral probing techniques -- such as "Would you lean more toward [answer] or [answer]?" or "Just your best guess is fine" -- to encourage valid responses. Probing can be particularly effective when respondents initially hesitate to provide an answer, and it can increase the number of valid responses.
However, some researchers have suggested that probing can lead to poor data quality because respondents may guess at an answer when they are asked a knowledge-based question for which there is a verifiably right or wrong answer. Probing also places an additional burden on the interviewer and can increase survey length.
To build on this research, Gallup conducted a series of experiments to study how probing affects responses to objective questions about the respondent as well as responses to attitudinal questions. Gallup also examined how probing affects data quality in different cultures and countries.
The results show that respondents who are probed after an initial "don't know/refused" response give systematically different answers than those who are not probed for a response. However, responses from probing make the data more complete and allow survey researchers to capture the opinions of those who would have otherwise been excluded. Still, researchers need to exercise caution when using probing on factual or knowledge-based questions and when questions consistently require probing.
How the Experiments Were Conducted
Gallup conducted experiments in the U.S. and Norway using the Gallup Daily tracking survey and the Gallup World Poll, respectively, to quantify the effects of probing. Gallup's standard approach is to have interviewers probe initial "don't know/refused" responses once, before accepting them as volunteered responses. During the experiments, Gallup asked interviewers to note whether they had to probe respondents. In the U.S., Gallup concentrated on specific factual and attitudinal questions; in Norway, interviewers noted whether they probed for every question asked in the survey.
The experiments were conducted in the U.S. June 16-Aug. 8, 2016, and in Norway in October 2016. In the U.S., 500 interviews per day (for a total of approximately 26,000 interviews) were analyzed for probing patterns. In Norway, about 460 interviews were analyzed.
In the U.S., after asking each of five specific survey questions, the computer system required interviewers to indicate whether they had probed the respondent. In Norway, interviewers noted whether they had probed after asking each question. See the Appendix at the bottom of this article for additional instructions provided to interviewers in Norway.
Financial Questions Probed Most Often in U.S. and Norway
In the U.S., the most commonly probed question among the five items was about whether respondents agreed or disagreed they were watching their spending closely. Although this question uses an agree/disagree scale, many respondents attempted to say yes or no, which was the main driver of the high probing rate.
The second most frequently probed question was presidential job approval. The high probing rate may have been due to the politically sensitive nature of the question, and the question coming first in the survey. It is also possible that Americans have complex views of a highly visible leader such as the president, and initial hesitation to provide a binary "approve/disapprove" answer could reflect difficulty in summarizing these attitudes.
|Do you agree or disagree with the following statement? You are watching your spending very closely.||40.7|
|Do you approve or disapprove of the way Barack Obama is handling his job as president?||23.1|
|Are you satisfied or dissatisfied with your standard of living, all the things you can buy and do?||15.3|
|Do you have any health problems that prevent you from doing any of the things people your age normally can do?||7.1|
|Would you say your own health, in general, is excellent, very good, good, fair or poor?||7.1|
Among the 100+ questions asked in Norway, the item probed most frequently is monthly household income. Household income is often perceived as a particularly personal and sensitive question, which contributes to the high nonresponse rate for this item. Most items requiring probing were open-ended questions.
|General Monthly Household Income||67.3|
|Number Friend Interactions||34.8|
|Number Years Formal Education||32.1|
|Specific Monthly Household Income||31.6|
|Number Hours Worked for Self||25.5|
|Ladder Question (Five Years)||20.9|
|Country Would Move to||20.5|
|Good, Affordable Housing||19.7|
|People Starting Businesses||19.3|
|Local Job Market||18.6|
|Number Hours Worked for Employer||18.0|
|Life Satisfaction on Ladder Scale (Today)||17.5|
|Deal With the Poor||17.3|
|Approval of U.S. Leadership||17.3|
|Approval of Germany's Leadership||15.5|
|Approval of Country's Leadership||15.2|
|Gallup World Poll|
In both the U.S. and Norway, less educated and older respondents were significantly more likely to require probing. However, there were no differences in the percentages probed by respondent gender or income.
Attitudes Among Probed Respondents Differ
To further determine the effects of probing, Gallup compared the final answers of respondents whom interviewers had to probe with those of respondents who provided valid responses immediately. These final answers were systematically different for those who were probed after an initial "don't know/refused" response. In the case of presidential job approval in the U.S., respondents who were probed were slightly less likely than those who were not probed to eventually say they approve of the president.
Similarly, those in the U.S. who were probed on a question about their health status were less likely than those who were not probed to report positive health ratings. While 36% of those who were probed eventually said their health was "excellent" or "very good," 52% of those who were not probed said the same. Although the percentage probed for each item is typically less than 50%, this group has a statistically significant effect on the final distribution of responses.
|Do you approve or disapprove of the way Barack Obama is handling his job as president?|
|Would you say your own health, in general, is excellent, very good, good, fair or poor?|
|Are you satisfied or dissatisfied with your standard of living, all the things you can buy and do?|
When respondents in Norway were asked whether they approve or disapprove of their country leadership, those who were probed were more likely than those who were not to eventually say they disapprove of their leadership. And when asked whether "it is now a good time or a bad time to find a job" in the city or area where they live, respondents who were probed were more likely to eventually say it is a good time to find a job in their area. The systematic differences in opinions among those who were probed on these items affected the final distribution of responses.
|Do you approve or disapprove of the job performance of the leadership of Norway?|
|Do you approve or disapprove of the job performance of the leadership of the U.S.A.?|
|Thinking about the job situation in the city or area where you live today, would you say that it is now a good time or a bad time to find a job?|
|Gallup World Poll|
Probing has long been regarded as an important part of interviewer-administered survey protocols. Gallup's protocols for probing are based on research that suggests that forms of nonresponse (don't know/refused) can be mitigated with effective interviewer probing. This is one of the key advantages of using interviewers over self-administered surveys.
Gallup's experiments show that respondents require probing on a variety of questions, including both knowledge-based and attitudinal questions. Importantly, the responses given by those who are probed are systematically different from those given by respondents who are not.
In summary, probing changes the final distribution of data. In addition to making the data more complete, it enhances the validity of our conclusions, as they are based on a more representative sample of individuals. The experiment results also show that probing may allow researchers to capture the opinions of individuals or non-random groups that would otherwise be excluded. Still, researchers should keep in mind that probed responses can be subject to higher error because respondents may be guessing at the answer or attempting to appease the interviewer.
Survey items that regularly require probing should be viewed with caution, as these are not primary volunteered responses. Researchers should consider whether each instance is a good use of probing (easing social pressure, or permitting an unpopular or private sentiment) or a bad use of probing (the question is genuinely confusing or too private, or one for which a respondent is being forced to guess). Researchers should pay attention to any item that requires consistent probing to see if modifications are needed to simplify or shorten the question, which would ease interviewer and respondent burden during the survey.
View the interviewer instructions and full item wording used in Norway in the Appendix.