One of the problems pollsters face as they design their questionnaires is the potential influence of question order. Last July, for example, I discussed why an ABC News/Washington Post poll found higher opposition than did Gallup to sending U.S. troops to Liberia. Unlike Gallup, the ABC/WP poll preceded its question on sending U.S. troops with a series of questions about the deteriorating situation in Iraq, which I suggested could have made respondents more cautious about another overseas mission.
Some colleagues have since challenged my interpretation, suggesting that instead the differences between the two polls more likely resulted from question wording (Gallup referred to "an international peace-keeping force," while ABC/WP used the phrase "an international force to help enforce a cease-fire"). The only way to resolve the different interpretations would be to conduct some split-sample experiments, in which the question wording is the same but the previous questions are different, or in which the previous questions are the same but the wording is different.
Such an experiment has not yet been conducted, at least by Gallup, but the importance of question order can be illustrated in another example. In an Oct. 10-12 poll, Gallup asked Americans how much they blamed the Israelis, and in a separate question, the Palestinians, for the current violence in the Middle East. Anticipating that question order could affect respondents' answers, we asked half the sample (Form A) about the Israelis and then about the Palestinians, while we asked the other half sample (Form B) about each group in reverse order.
|
FORM A: Questions asked in order below |
|
FORM B: Questions asked in reverse order. |
|
How much do you blame the Israelis for the current violence in the Middle East -- a great deal, a moderate amount, not very much or not at all? |
|
How much do you blame the Israelis for the current violence in the Middle East -- a great deal, a moderate amount, not very much or not at all? |
The results of the experiment show a significant difference between the two forms in the percentage who blame the Israelis. When the Israelis were mentioned first, just 62% said they are to blame "a great deal" or "a moderate amount." When the Israelis were mentioned after the question about the Palestinians, 74% said the Israelis are to blame "a great deal" or "a moderate amount."
|
How Much Israelis Are to Blame for Violence in the Middle East |
|||
|
Form A |
Form B |
Difference |
|
|
% |
% |
Pct. Pts. |
|
|
Great deal/moderate amount |
62 |
74 |
+12 |
|
Not very much/not at all |
34 |
23 |
-11 |
|
No opinion |
4 |
3 |
-1 |
Such is not the case for the Palestinians, however. When respondents were asked first how much the Palestinians are to blame (Form B), 82% said a great deal or moderate amount. When asked about the Palestinians after a similar question about the Israelis (Form A), almost the same percentage, 79%, as in Form B said a great deal or moderate amount.
|
How Much Palestinians Are to Blame for Violence in the Middle East |
|||
|
Form A |
Form B |
Difference |
|
|
% |
% |
Pct. Pts. |
|
|
Great deal/moderate amount |
79 |
82 |
+3 |
|
Not very much/not at all |
17 |
14 |
-3 |
|
No opinion |
4 |
4 |
0 |
These results show that question order makes a significant difference in the percentage who blame the Israelis. Many more people are willing to blame the Israelis after they have first been asked a blame question about the Palestinians than when they are asked about the Israelis without a similar question about the Palestinians.
Both measures of Israeli blame give insights into what Americans are thinking, but is there one measure that is more valid than the other? One answer is that both answers are equally valid. One measure suggests how Americans view Palestinians or Israelis when each group is considered alone. The other measure suggests how Americans view the two groups when both are considered simultaneously.
The major point here is that the order in which questions are asked can influence the responses that people give. These "question context" effects are often unavoidable -- as in this case. Because we wanted to ask about both groups, clearly one had to go before the other.
To minimize the context effects, we rotated question order. And in reporting the data, Gallup averaged the results of the two forms, giving the following results: 68% of Americans said Israelis were to blame a great deal or moderate amount, and 81% said Palestinians were to blame by that amount. As with all summary data, these results represent the best single-point estimates of the results, but also conceal other estimates that reveal additional complexities to public opinion.