The Effects of Probing in Survey Research

Probing is a common technique that researchers use in interviewer-administered surveys when respondents initially refuse to answer a question or say they "don't know." Interviewers are trained to use neutral probing techniques -- such as "Would you lean more toward [answer] or [answer]?" or "Just your best guess is fine" -- to encourage valid responses. Probing can be particularly effective when respondents initially hesitate to provide an answer, and it can increase the number of valid responses.

However, some researchers have suggested that probing can lead to poor data quality because respondents may guess at an answer when they are asked a knowledge-based question for which there is a verifiably right or wrong answer. Probing also places an additional burden on the interviewer and can increase survey length.

To build on this research, Gallup conducted a series of experiments to study how probing affects responses to objective questions about the respondent as well as responses to attitudinal questions. Gallup also examined how probing affects data quality in different cultures and countries.

The results show that respondents who are probed after an initial "don't know/refused" response give systematically different answers than those who are not probed for a response. However, responses from probing make the data more complete and allow survey researchers to capture the opinions of those who would have otherwise been excluded. Still, researchers need to exercise caution when using probing on factual or knowledge-based questions and when questions consistently require probing.

How the Experiments Were Conducted

Gallup conducted experiments in the U.S. and Norway using the Gallup Daily tracking survey and the Gallup World Poll, respectively, to quantify the effects of probing. Gallup's standard approach is to have interviewers probe initial "don't know/refused" responses once, before accepting them as volunteered responses. During the experiments, Gallup asked interviewers to note whether they had to probe respondents. In the U.S., Gallup concentrated on specific factual and attitudinal questions; in Norway, interviewers noted whether they probed for every question asked in the survey.

The experiments were conducted in the U.S. June 16-Aug. 8, 2016, and in Norway in October 2016. In the U.S., 500 interviews per day (for a total of approximately 26,000 interviews) were analyzed for probing patterns. In Norway, about 460 interviews were analyzed.

In the U.S., after asking each of five specific survey questions, the computer system required interviewers to indicate whether they had probed the respondent. In Norway, interviewers noted whether they had probed after asking each question. See the Appendix at the bottom of this article for additional instructions provided to interviewers in Norway.

Financial Questions Probed Most Often in U.S. and Norway

In the U.S., the most commonly probed question among the five items was about whether respondents agreed or disagreed they were watching their spending closely. Although this question uses an agree/disagree scale, many respondents attempted to say yes or no, which was the main driver of the high probing rate.

The second most frequently probed question was presidential job approval. The high probing rate may have been due to the politically sensitive nature of the question, and the question coming first in the survey. It is also possible that Americans have complex views of a highly visible leader such as the president, and initial hesitation to provide a binary "approve/disapprove" answer could reflect difficulty in summarizing these attitudes.

 

	Probed
	%
Survey Item
Do you agree or disagree with the following statement? You are watching your spending very closely.	40.7
Do you approve or disapprove of the way Barack Obama is handling his job as president?	23.1
Are you satisfied or dissatisfied with your standard of living, all the things you can buy and do?	15.3
Do you have any health problems that prevent you from doing any of the things people your age normally can do?	7.1
Would you say your own health, in general, is excellent, very good, good, fair or poor?	7.1
Gallup

 

 

Among the 100+ questions asked in Norway, the item probed most frequently is monthly household income. Household income is often perceived as a particularly personal and sensitive question, which contributes to the high nonresponse rate for this item. Most items requiring probing were open-ended questions.

 

	Probed
	%
Survey Item
General Monthly Household Income	67.3
Education Level	45.7
Number Friend Interactions	34.8
Religion	33.6
Number Years Formal Education	32.1
Specific Monthly Household Income	31.6
Number Hours Worked for Self	25.5
Ladder Question (Five Years)	20.9
Country Would Move to	20.5
Good, Affordable Housing	19.7
People Starting Businesses	19.3
Local Job Market	18.6
Marital Status	18.6
Number Hours Worked for Employer	18.0
Life Satisfaction on Ladder Scale (Today)	17.5
Deal With the Poor	17.3
Approval of U.S. Leadership	17.3
Re-contact	15.9
Approval of Germany's Leadership	15.5
Approval of Country's Leadership	15.2
Gallup World Poll

 

 

In both the U.S. and Norway, less educated and older respondents were significantly more likely to require probing. However, there were no differences in the percentages probed by respondent gender or income.

Attitudes Among Probed Respondents Differ

To further determine the effects of probing, Gallup compared the final answers of respondents whom interviewers had to probe with those of respondents who provided valid responses immediately. These final answers were systematically different for those who were probed after an initial "don't know/refused" response. In the case of presidential job approval in the U.S., respondents who were probed were slightly less likely than those who were not probed to eventually say they approve of the president.

Similarly, those in the U.S. who were probed on a question about their health status were less likely than those who were not probed to report positive health ratings. While 36% of those who were probed eventually said their health was "excellent" or "very good," 52% of those who were not probed said the same. Although the percentage probed for each item is typically less than 50%, this group has a statistically significant effect on the final distribution of responses.

 

	Probed	Not probed
	%	%
Do you approve or disapprove of the way Barack Obama is handling his job as president?
Approve	48	51
Disapprove	37	48
Would you say your own health, in general, is excellent, very good, good, fair or poor?
Excellent	14	20
Very good	22	32
Good	37	29
Fair	21	15
Poor	5	5
Are you satisfied or dissatisfied with your standard of living, all the things you can buy and do?
Satisfied	65	81
Dissatisfied	32	19
Gallup

 

 

When respondents in Norway were asked whether they approve or disapprove of their country leadership, those who were probed were more likely than those who were not to eventually say they disapprove of their leadership. And when asked whether "it is now a good time or a bad time to find a job" in the city or area where they live, respondents who were probed were more likely to eventually say it is a good time to find a job in their area. The systematic differences in opinions among those who were probed on these items affected the final distribution of responses.

 

	Probed	Not probed
	%	%
Do you approve or disapprove of the job performance of the leadership of Norway?
Approve	56	81
Disapprove	35	18
Do you approve or disapprove of the job performance of the leadership of the U.S.A.?
Approve	48	62
Disapprove	31	38
Thinking about the job situation in the city or area where you live today, would you say that it is now a good time or a bad time to find a job?
Good time	59	38
Bad time	29	60
Gallup World Poll

 

 

Conclusion

Probing has long been regarded as an important part of interviewer-administered survey protocols. Gallup's protocols for probing are based on research that suggests that forms of nonresponse (don't know/refused) can be mitigated with effective interviewer probing. This is one of the key advantages of using interviewers over self-administered surveys.

Gallup's experiments show that respondents require probing on a variety of questions, including both knowledge-based and attitudinal questions. Importantly, the responses given by those who are probed are systematically different from those given by respondents who are not.

In summary, probing changes the final distribution of data. In addition to making the data more complete, it enhances the validity of our conclusions, as they are based on a more representative sample of individuals. The experiment results also show that probing may allow researchers to capture the opinions of individuals or non-random groups that would otherwise be excluded. Still, researchers should keep in mind that probed responses can be subject to higher error because respondents may be guessing at the answer or attempting to appease the interviewer.

Survey items that regularly require probing should be viewed with caution, as these are not primary volunteered responses. Researchers should consider whether each instance is a good use of probing (easing social pressure, or permitting an unpopular or private sentiment) or a bad use of probing (the question is genuinely confusing or too private, or one for which a respondent is being forced to guess). Researchers should pay attention to any item that requires consistent probing to see if modifications are needed to simplify or shorten the question, which would ease interviewer and respondent burden during the survey.

Read more from Gallup's Methodology Blog and visit our Methodology Center.

View the interviewer instructions and full item wording used in Norway in the Appendix.