skip to main content

The Effects of Probing in Survey Research

The Effects of Probing in Survey Research
by Stephanie Marken and Sofia Kluch

Probing is a common technique that researchers use in interviewer-administered surveys when respondents initially refuse to answer a question or say they "don't know." Interviewers are trained to use neutral probing techniques -- such as "Would you lean more toward [answer] or [answer]?" or "Just your best guess is fine" -- to encourage valid responses. Probing can be particularly effective when respondents initially hesitate to provide an answer, and it can increase the number of valid responses.

However, some researchers have suggested that probing can lead to poor data quality because respondents may guess at an answer when they are asked a knowledge-based question for which there is a verifiably right or wrong answer. Probing also places an additional burden on the interviewer and can increase survey length.

To build on this research, Gallup conducted a series of experiments to study how probing affects responses to objective questions about the respondent as well as responses to attitudinal questions. Gallup also examined how probing affects data quality in different cultures and countries.

The results show that respondents who are probed after an initial "don't know/refused" response give systematically different answers than those who are not probed for a response. However, responses from probing make the data more complete and allow survey researchers to capture the opinions of those who would have otherwise been excluded. Still, researchers need to exercise caution when using probing on factual or knowledge-based questions and when questions consistently require probing.

How the Experiments Were Conducted

Gallup conducted experiments in the U.S. and Norway using the Gallup Daily tracking survey and the Gallup World Poll, respectively, to quantify the effects of probing. Gallup's standard approach is to have interviewers probe initial "don't know/refused" responses once, before accepting them as volunteered responses. During the experiments, Gallup asked interviewers to note whether they had to probe respondents. In the U.S., Gallup concentrated on specific factual and attitudinal questions; in Norway, interviewers noted whether they probed for every question asked in the survey.

The experiments were conducted in the U.S. June 16-Aug. 8, 2016, and in Norway in October 2016. In the U.S., 500 interviews per day (for a total of approximately 26,000 interviews) were analyzed for probing patterns. In Norway, about 460 interviews were analyzed.

In the U.S., after asking each of five specific survey questions, the computer system required interviewers to indicate whether they had probed the respondent. In Norway, interviewers noted whether they had probed after asking each question. See the Appendix at the bottom of this article for additional instructions provided to interviewers in Norway.

Financial Questions Probed Most Often in U.S. and Norway

In the U.S., the most commonly probed question among the five items was about whether respondents agreed or disagreed they were watching their spending closely. Although this question uses an agree/disagree scale, many respondents attempted to say yes or no, which was the main driver of the high probing rate.

The second most frequently probed question was presidential job approval. The high probing rate may have been due to the politically sensitive nature of the question, and the question coming first in the survey. It is also possible that Americans have complex views of a highly visible leader such as the president, and initial hesitation to provide a binary "approve/disapprove" answer could reflect difficulty in summarizing these attitudes.

Percentage of U.S. Respondents Requiring Probing, by Survey Item
Probed
%
Survey Item
Do you agree or disagree with the following statement? You are watching your spending very closely. 40.7
Do you approve or disapprove of the way Barack Obama is handling his job as president? 23.1
Are you satisfied or dissatisfied with your standard of living, all the things you can buy and do? 15.3
Do you have any health problems that prevent you from doing any of the things people your age normally can do? 7.1
Would you say your own health, in general, is excellent, very good, good, fair or poor? 7.1
Gallup

Among the 100+ questions asked in Norway, the item probed most frequently is monthly household income. Household income is often perceived as a particularly personal and sensitive question, which contributes to the high nonresponse rate for this item. Most items requiring probing were open-ended questions.

Percentage of Norway Respondents Requiring Probing, by Survey Item
Probed
%
Survey Item
General Monthly Household Income 67.3
Education Level 45.7
Number Friend Interactions 34.8
Religion 33.6
Number Years Formal Education 32.1
Specific Monthly Household Income 31.6
Number Hours Worked for Self 25.5
Ladder Question (Five Years) 20.9
Country Would Move to 20.5
Good, Affordable Housing 19.7
People Starting Businesses 19.3
Local Job Market 18.6
Marital Status 18.6
Number Hours Worked for Employer 18.0
Life Satisfaction on Ladder Scale (Today) 17.5
Deal With the Poor 17.3
Approval of U.S. Leadership 17.3
Re-contact 15.9
Approval of Germany's Leadership 15.5
Approval of Country's Leadership 15.2
Gallup World Poll

In both the U.S. and Norway, less educated and older respondents were significantly more likely to require probing. However, there were no differences in the percentages probed by respondent gender or income.

##SPEEDBUMP##

Attitudes Among Probed Respondents Differ

To further determine the effects of probing, Gallup compared the final answers of respondents whom interviewers had to probe with those of respondents who provided valid responses immediately. These final answers were systematically different for those who were probed after an initial "don't know/refused" response. In the case of presidential job approval in the U.S., respondents who were probed were slightly less likely than those who were not probed to eventually say they approve of the president.

Similarly, those in the U.S. who were probed on a question about their health status were less likely than those who were not probed to report positive health ratings. While 36% of those who were probed eventually said their health was "excellent" or "very good," 52% of those who were not probed said the same. Although the percentage probed for each item is typically less than 50%, this group has a statistically significant effect on the final distribution of responses.

Final Answers to Select Items Among U.S. Respondents, by Probing Status
Probed Not probed
% %
Do you approve or disapprove of the way Barack Obama is handling his job as president?
Approve 48 51
Disapprove 37 48
Would you say your own health, in general, is excellent, very good, good, fair or poor?
Excellent 14 20
Very good 22 32
Good 37 29
Fair 21 15
Poor 5 5
Are you satisfied or dissatisfied with your standard of living, all the things you can buy and do?
Satisfied 65 81
Dissatisfied 32 19
Gallup

When respondents in Norway were asked whether they approve or disapprove of their country leadership, those who were probed were more likely than those who were not to eventually say they disapprove of their leadership. And when asked whether "it is now a good time or a bad time to find a job" in the city or area where they live, respondents who were probed were more likely to eventually say it is a good time to find a job in their area. The systematic differences in opinions among those who were probed on these items affected the final distribution of responses.

Final Answers to Select Items Among Norway Respondents, by Probing Status
Probed Not probed
% %
Do you approve or disapprove of the job performance of the leadership of Norway?
Approve 56 81
Disapprove 35 18
Do you approve or disapprove of the job performance of the leadership of the U.S.A.?
Approve 48 62
Disapprove 31 38
Thinking about the job situation in the city or area where you live today, would you say that it is now a good time or a bad time to find a job?
Good time 59 38
Bad time 29 60
Gallup World Poll

Conclusion

Probing has long been regarded as an important part of interviewer-administered survey protocols. Gallup's protocols for probing are based on research that suggests that forms of nonresponse (don't know/refused) can be mitigated with effective interviewer probing. This is one of the key advantages of using interviewers over self-administered surveys.

Gallup's experiments show that respondents require probing on a variety of questions, including both knowledge-based and attitudinal questions. Importantly, the responses given by those who are probed are systematically different from those given by respondents who are not.

In summary, probing changes the final distribution of data. In addition to making the data more complete, it enhances the validity of our conclusions, as they are based on a more representative sample of individuals. The experiment results also show that probing may allow researchers to capture the opinions of individuals or non-random groups that would otherwise be excluded. Still, researchers should keep in mind that probed responses can be subject to higher error because respondents may be guessing at the answer or attempting to appease the interviewer.

Survey items that regularly require probing should be viewed with caution, as these are not primary volunteered responses. Researchers should consider whether each instance is a good use of probing (easing social pressure, or permitting an unpopular or private sentiment) or a bad use of probing (the question is genuinely confusing or too private, or one for which a respondent is being forced to guess). Researchers should pay attention to any item that requires consistent probing to see if modifications are needed to simplify or shorten the question, which would ease interviewer and respondent burden during the survey.

Read more from Gallup's Methodology Blog and visit our Methodology Center.

View the interviewer instructions and full item wording used in Norway in the Appendix.

Gallup


Gallup https://news.gallup.com/opinion/methodology/223286/effects-probing-survey-research.aspx
Gallup World Headquarters, 901 F Street, Washington, D.C., 20001, U.S.A
+1 202.715.3030