The BBC recently released a major study of public opinion across 21 nations (and a separate poll of the U.S., bringing the total to 22 nations). The poll focused on attitudes toward the U.S. and President Bush.
Here's a sample of how the results of the poll were analyzed, as presented on the Web site of the Program on International Policy Attitudes at the University of Maryland, a partner with the BBC World Service in conducting the poll:
In 18 of 21 Countries Polled, Most See Bush's Reelection as Negative for World Security
India, Philippines and Poland See Bush's Reelection as Positive
Substantial Minority Now Feels Worse Toward American People
More See US Influence in World as Negative Than Positive
No Country Supports Contributing Troops to Iraq
According to a new BBC World Service Poll of twenty-one countries from all regions of the world, the reelection of President Bush is seen as negative for world peace and security by a majority in sixteen countries and a plurality in another two.
On average across all countries, 58 percent said that Bush's reelection was negative, while 26 percent said that it was positive for global security.
It's an interesting poll, and certainly deals with very important issues. I have a lot of respect not only for the BBC, but for the PIPA program at Maryland (headed by scholar Dr. Stephen Kull). As far as we can tell, the surveys appear to have been conducted (by various firms across the 22 countries) with appropriate sample sizes, and presumably using scientific random sampling techniques, although that is difficult to determine from the release.
Still, I think there are a couple of points worth making in terms of analyzing the results of the poll. I don't think these points change the overall interpretation of the data, but they are issues with which users of the reported data should be familiar.
First, it is important to note that in eight of the countries in which interviews were conducted (Brazil, Chile, China, India, Indonesia, Philippines, South Africa, and Turkey), the interviews were in selected urban cities only and not across the entire country. The countries involved in this list are among the largest included in the sample. So it's important to point out that the headline most probably should have read: "In 18 of 21 Countries or Major Cities Within Countries Polled, Most See Bush's Reelection as Negative for World Security."
It's unclear how different the data would have been if the entire population in each of these eight countries had been sampled. One could advance an argument that on issues such as these, the views of rural residents would be less important, assuming that they are less in touch with what is happening in the world.
But my point is one of precision in discussing poll results. Regardless of the rationale involved, it is important to let the reader or user of the data know the precise nature of the populations being sampled.
There's a second issue with the report of the BBC poll with which we are very familiar here at Gallup. That's the decision on how to weight data across countries.
Note that the main headline in the PIPA report presented above doesn't use an average figure, but specifies "Within 18 of 21 countries …", etc. That's fine.
But the subhead farther down says: "On average across all countries, 58 percent said that Bush's reelection was negative, while 26 percent said that it was positive for global security." (The reports of the poll on the BBC also featured the 26% figure, as did many secondary news reports.)
That's where the averaging issue comes in.
The report's authors arrive at this 26% Bush positive/58% negative average by adding up the figures for each country and dividing by the total number of countries. In other words, each country received exactly the same weight in computing these averages. Lebanon (with a population of less than 4 million) received the same weight in this world average as did India (with a population of more than 1 billion).
Clearly if one is talking about the average response of the people living in the 21 countries involved here, which I think is the implication, one should weight the data according to the population of each country. If the report authors are truly talking about a "world" poll, then they need to represent the people of the world appropriately. In fact, when that is done, an estimate by Gallup analysis suggests that the numbers are 35% Bush positive, 49% Bush negative. That doesn't change the interpretation dramatically, but it is more accurate.
The procedure used in the average as presented would be akin to measuring Bush job approval in 50 polls across the 50 states in the U.S., and then giving each state's average job approval rating an equal weight in computing a U.S. average. Using such a procedure, Bush's job approval rating would be higher than it is in a proportionate national poll, because there are more red states in the country than blue states, and giving each state an equal weight would therefore give more relative weight to red states than would a proportionate poll. That is, red states with small populations such as Wyoming, Nebraska, Utah, and so forth would receive the same weight as the more populous blue states such as California and New York.
That's not to say that there might not be some situations in which one would want to weight by entity rather than by population within entity, but that type of rationale would need to be made crystal clear in the analysis and reporting.
I said above that we here in Gallup are quite familiar with this issue. When we released a poll of nine predominantly Islamic countries after 9/11, our initial report also included graphs that had an unweighted average across the nine countries. Some people picked up on that average and called our attention to the fact that it could cause misunderstanding. So we changed the graphs and removed averages altogether, referring only to the countries individually.