skip to main content
Rotating Responses Offer Insights Into the Nature of Public Opinion

Rotating Responses Offer Insights Into the Nature of Public Opinion

by David W. Moore
Last week, I described a Los Angeles Times poll question that presented two arguments about tax cuts -- one positive and one negative. I said it appeared that all respondents had been read the arguments in the same order, with the negative one last. Susan Pinkus, director of the poll, sent me an e-mail saying the arguments were rotated, a fact that is noted on the poll's Web site. I should have checked the original source and apologize for the incorrect information. (She also said she believed her question was "balanced," while I said it was slightly biased. But that is a subjective judgment, which readers can decide for themselves.)

Why Rotate?

Which leads me to discuss why pollsters rotate responses in the first place. In everyday conversations, people are probably unaware of any bias that might occur because of the order in which they present two choices for consideration. For example, if someone asks you, "Would you rather go to the movie now or get something to eat first?", your answer would probably be the same as if that person asked, "Would you rather get something to eat first or go to the movie now?"

But research has shown that the order in which the two choices are presented could affect how some people respond. The structure of the question would not influence most people, but for a small number of people who are ambivalent about going to the movie or eating first, the order in which the options are presented could influence their responses.

Take this example from a Gallup Poll in 1998:

As you may have heard, a number of black churches have been destroyed by fire in the United States over the past several years -- and authorities have attributed most of these to arson.

1. Some people say these incidents reflect the fact that American society is basically racist.

2. Other people say these are isolated incidents that do not reflect on society as a whole.

Which comes closer to your point of view?

Half the sample (Form A) was read the question in the format shown in the text above, but for the other half sample (Form B), the arguments were read in reverse order.

Why Are Black Churches Being Destroyed by Fire?

 

 

Form A

Overall

Form B

 

%

%

%

       

America is a racist society

45

49

54

These are isolated incidents

51

44

39

       

March 20-22, 1998

     


When the "isolated incidents" option was read last (Form A), a majority chose it -- by 51% to 45%. However, when the options were reversed (Form B) and "America is a racist society" was read last, it received a majority -- by 54% to 39%. Gallup actually reported the overall the results, indicating that on average, 49% of respondents felt incidents of arson were a reflection of a racist society, while 44% said they were isolated incidents.

The order in which the responses were read did not affect most people, but some people held their opinions so lightly that the slight change in the question structure influenced them -- they were more likely to choose the option that was read last than the one that was read first.

A more recent example comes from a March 3-5 poll this year. The question asked:

With which one of these statements about the environment and energy production do you most agree – [Form A: Read 1-2; Form B: Read 2-1]?

1. Protection of the environment should be given priority, even at the risk of limiting the amount of energy supplies -- such as oil, gas and coal -- which the United States produces (or)

2. Development of U.S. energy supplies -- such as oil, gas and coal -- should be given priority, even if the environment suffers to some extent (or)

More respondents chose "protection of the environment" in each form, but the margin was 16 percentage points in Form A (52% to 36%), but just three percentage points in Form B (47% to 44%). In this case, when an option was read first, it was a little more likely to be chosen than when it was read second.

Priority of Environment vs. Energy Supplies

 

 

 

Form A

Overall

Form B

 

%

%

%

       

Protection of environment

52

49

47

Development of energy supplies

36

40

44

       

March 3-5, 2003

     


The results of the two forms would have led to quite different conclusions -- that a substantial majority prefers protection of the environment (Form A), or that the public is almost evenly divided (Form B). What is the "truth"? Clearly, the public at least leans toward protection of the environment over the development of energy supplies, but for many Americans the issue is not firmly settled -- suggesting that their opinions could easily change as they learn more about the issue.

Polling experience shows that on most questions, the order of the responses does not matter. And when there are differences, they tend to be small. But pollsters will rotate the responses anyway, to minimize the potential impact. When we find large differences because of the question rotation, it suggests an issue that may be especially volatile -- one in which even minor changes in information may have a large impact on what the public is thinking.


Gallup https://news.gallup.com/poll/8578/rotating-responses-offer-insights-into-nature-public-opinion.aspx
Gallup World Headquarters, 901 F Street, Washington, D.C., 20001, U.S.A
+1 202.715.3030