PRINCETON, NJ -- The 2002 Gallup Poll of the Islamic World has raised interesting questions about measurement in survey research and social science.
We have received inquires about the degree to which the attitudes we measured in these predominantly Islamic countries reflect access to certain government-controlled media. Some writers tell us that the respondents' attitudes reflect only their country's "official" position. Along these same lines is the question about whether or not the respondents are "telling the truth." One journalist with whom I appeared on a television program observed that polls can never be trusted in many of these predominantly Islamic countries because the people are somehow afraid to say what they really think.
These are important questions that we also hear in relationship to our polls conducted in the United States. They seem to come down to a very general and basic question: Just what is it that we are measuring in polls?
Well, for one thing, we have learned over the years that the concept of a "true" attitude isn't necessarily a particularly useful one. People don't have ironclad, sharply defined, rationally based, immutable views on every topic or issue that they file away in a lock-box in the interior of their brain -- waiting for the shrewd interviewer to tap.
Instead, people have the propensity to behave toward specific stimuli in certain evaluative ways. These tendencies are analogous to probabilities. To say that someone in this country is "anti-abortion" is to imply that they have a high probability of responding in negative ways toward the concept of abortion. But the exact nature and value of this response can vary depending on things such as the way a question is phrased and the environment in which it is asked.
As of now, there is no way of reliably measuring some hypothetical, basic, or true attitude lodged "in the brain" (although complex medical imaging technology can identify how a respondent's neurons fire when exposed to certain stimuli.) We have to rely on verbal and written expressions.
This leads us to say that responses to the questions asked in the nine Islamic countries included in the poll represent tendencies to react in certain ways to specific stimuli, measured by an interviewer in an in-home environment. The responses reflect the background, socialization, media exposure, and life experiences of the respondents. And they reflect, at least to some degree, the environment in which they are measured.
So, it is most certainly true that the official government stance of the countries in which the surveys were conducted is reflected, at least in part, in the responses we obtained. And, it is true that the interview environment may have had some impact on the exact nature of these responses. But none of this makes the responses invalid. Whatever their origin, the expressed attitudes reflect reality as of the day of the survey interviews. Despite the possible impact of the environment in which they were measured, they still reflect tendencies that are likely to be expressed in other situations.
Let's look at this last point. We assume that when we conduct interviews, what is measured in the interview setting bears a strong relationship to the responses to the same types of issues we would find if we could follow the respondents around after the interview and continue to observe their behavior. This is the so-called "attitude-behavior" link that many social scientists have examined for years. It's really a "behavior-behavior" link; that is, the link between a verbal behavior expressed in an interview setting and some other behavior expressed in some other setting.
When we have a chance to measure this type of relationship in the United States, say with pre-election polls, we usually find it is quite strong. How people react to candidates in polls is usually quite powerfully related to how they behave in the voting booth.
We don't have a specific behavioral standard against which to relate the attitudes we measured in the polls in nine predominantly Islamic countries. So, we don't know precisely the degree to which what the respondents told us in the interview setting may or may not be related to what they may say or do in other settings.
But, there are a couple of specific points to be made. First, we set up an interview situation specifically designed to provide a neutral, comfortable environment in which the responses can be measured. Most of the time, in fact, an interview setting is usually a pretty good situation in which to tap into basic attitudinal tendencies, in large part because it is designed to be neutral.
Each interview conducted in the nine Islamic countries was about one hour in length. It included, very deliberately, a variety of questions about media use, personal striving, values in life, and so forth. Questions that pertained to the United States, Sept. 11, or the war in Afghanistan were combined as parts of lists so that they did not stand out as the sole purpose of the interview. Many of the questions were not controversial. The interviews were conducted by indigenous, experienced market research interviewers who conduct interviews in these countries as a matter of course. In many of these countries, in fact, surveys are fairly common and familiar to people. Everything we know about the situation suggests that the interview setting was a neutral one in which the respondents felt comfortable to express their opinions.
There is also a second and subtler point. Assume for the moment that one of the reasons we obtained these responses was because the respondent was giving an "expected" answer based on his or her perception of the standard, government opinion. This doesn't vitiate the impact of the answers. These responses still have great meaning. If these responses are not the "true" attitudes of the people being interviewed, what are they? They are given for a reason. If respondents are compelled to say one thing to an interviewer that might be different from what is said in another (that is, a more private) situation, then we have evidence for the existence of very powerful norms in his or her respective society. This is an important finding. In other words, regardless of the relationship between the expressed attitudes and what might be expressed in other situations (say, to family or friends), the fact that the respondents feel compelled to express the attitudes we measured in the interview situation remains very important. And, again, without further evidence or research, our starting-point assumption is that these expressions as measured in the interview reflect basic tendencies that we would find expressed by respondents in other situations.
Now, let's look at this in a slightly different way. We assume that these responses, in some way, reflect official government sources of information and/or some type of implicit or explicit pressure from government, friends, and businesses to express certain attitudes. Everyone's attitudes must come from some source -- usually a combination of early socialization, family, peers, media exposure, and even genetic tendencies. There is no doubt that official government policies and communications can affect them.
But does this fact of life make the responses any less valid? Do we dismiss them because they have not been formed in some perfectly pure and deliberative way? No. These attitudes exist at the time of the interview and that is what we measure, regardless of their origin. The implication of some of the comments we have heard is that these expressions mean less if they simply reflect the official government line. But, the very fact that they reflect the government is obviously important in and of itself.
Indeed, those who use the results of the nine-country poll of the Islamic world may want to investigate the origins of the attitudes further. Certainly, the Bush administration is convinced that it is important to supply the residents of these nine countries with additional information in order to change these attitudes. It may well be that these people would feel more positively toward the United States if they met Americans, were able to follow American media closely, had long seminars with U.S. government officials, and otherwise got out from under their informational umbrella. But until that point, the attitudes are as we find them. If they change, we will measure them when we conduct follow-up research. But for now, saying that the attitudes reported are a result of government-controlled media does not in and of itself make them any less valid.
Of course, we shouldn't overlook the fact that many of these attitudes don't reflect the government policies in some countries. Obviously, there is some independence of thought among the population when the government of at least one of the countries included in the study has gone to a great deal of trouble to try to discredit the results of the poll.
The current research documents the way in which over 10,000 representatively selected people living in these nine countries respond to specific questions, and provides a valuable record of where people in these nine countries stand on these issues as of today.