skip to main content
Government an Immediate Threat? Question Wording Makes a Difference

Government an Immediate Threat? Question Wording Makes a Difference

by David W. Moore
Any frequent reader of polls knows that the way a question is worded can influence how people respond. This is one example of how slight differences in question wording led to major differences in how people characterized their government.

In late April 1995, just days after the Oklahoma City bombing, Gallup asked Americans if they thought the government had become so large and powerful that it posed an immediate threat to the rights and freedoms of ordinary citizens. We were shocked to find that 4 in 10 (39%) said yes.

The phrasing of the question reflected political commentary at the time that suggested many people felt the government had become so big that it was a danger to its own citizens. We wanted to discover how many Americans shared that opinion. But when we got the results, we wondered if including the phrase, "has become so large and powerful," may have distorted the results by subtly encouraging people to say there was indeed a threat.

In a March 1996 poll, we asked the same question of half the sample, but of the other half sample, we asked a different version.

Original Wording (asked of half sample): Do you think the federal government has become so large and powerful that it poses an immediate threat to the rights and freedoms of ordinary citizens, or don't you think so?


Revised Wording (asked of other half sample): In your opinion, is the federal government in Washington an immediate threat to the rights and freedoms of ordinary citizens like yourself, or not?


When the original wording was used, the poll found 50% of Americans saying the government was an immediate threat, but only 31% saying an immediate threat when the revised wording was used. Given the split-sample design, we can be confident that the substantial 19-point difference was due to question wording differences.

Question Wording Differences

Unfortunately, the revised question included two substantive changes, so we could not be sure how much of the 19-point difference in results was caused by each change.

The revised question did not include the phrase that said the federal government "has become so large and powerful" that it poses an immediate threat. Without that phrase, we would expect fewer people to conclude the government was a threat.

But the revised version added a short phrase referring to the rights and freedoms of ordinary citizens "like yourself." While people may believe there are threats to society, they are less likely to say that the threats apply to themselves personally.

(There were also three minor changes, which we are confident had no impact on the results. The original version started by saying, "Do you think . . . ", while the revised version said, "In your opinion . . . ". Also, the original version said at the end, "or don't you think so?", while the revised version said, "or not?" Finally, the revised version referred to the federal government "in Washington," while the original version mentioned no location.)

Potential Impact of Question Wording Differences

In the latest Gallup Poll, we again included the original version of the question, but this time included a different alternate version.

Revised Wording #2: Do you think the federal government poses an immediate threat to the rights and freedoms of ordinary citizens, or not?


The only substantive difference between this version and the original one is that this one does not include the "large and powerful" phrase. Also, this one -- unlike the first revised version -- does not refer to ordinary citizens "like yourself."

The current poll found that the original version shows 39% of Americans saying the government is an immediate threat (the same as in 1995), compared with 30% of Americans who take that view when asked the revised wording. Thus, we can conclude that when respondents hear the phrase about the government becoming "so large and powerful," they are more likely to conclude there is a threat than when the phrase is not included.

But the nine-point difference we measured this time is about half of the 19-point difference we found in 1996, when we used Revised Wording #1 (which included the phrase, "citizens like yourself"). From these results, we might assume that both changes -- not mentioning the size of government, as well as adding the phrase "like yourself" -- have a substantial impact on how people characterize their government. But we can't be sure until we conduct a more elaborate experiment to isolate the individual impact of each change simultaneously.

This example illustrates how influential slight variations in question wording may be on polling results. It also shows that regardless of question wording, a substantial proportion of Americans feel the federal government is an "immediate" threat to the rights and freedoms of ordinary citizens. But interpreting what people mean by that response is a challenge by itself -- which I will address in a future article.


Gallup https://news.gallup.com/poll/9250/government-immediate-threat-question-wording-makes-difference.aspx
Gallup World Headquarters, 901 F Street, Washington, D.C., 20001, U.S.A
+1 202.715.3030