WASHINGTON, D.C. -- In March 2024, the Office of Management and Budget (OMB) released updated standards for collecting self-reported race and ethnicity from respondents on U.S. federal government surveys. This was the first revision since 1997. These changes will influence how federal agencies, including the Census Bureau, ask about and report on race and ethnicity.
While the OMB standards only officially apply to government data collection, the changes will affect researchers seeking to understand or measure attitudes and behaviors by race and ethnicity or who weight U.S. survey data using federal data sources. Many organizations and research projects outside of the federal government are now evaluating when they should make their own changes to the questions and how they may affect estimates of race and ethnicity.
Gallup conducted a large survey experiment in spring 2024 to understand how the updates to the standards could affect responses. Gallup ultimately found that the approach to collecting race and ethnicity matters, and particularly will affect the estimates of Hispanic White and multiracial respondents.
This article outlines the changes to the updated standards, Gallup’s experiment and findings.
Background on OMB Changes
Before the revisions, the standards separated Hispanic ethnicity or origin and race information into two questions. The first question asked about Hispanic ethnicity, while the second question asked about race and included five required categories: White, Black or African American, Asian, American Indian or Alaska Native, and Native Hawaiian or Pacific Islander.
Previous research found that separating race and Hispanic ethnicity into separate questions can confuse some respondents, especially Hispanic individuals who may not identify with the existing racial categories recognized by OMB (for example, White, Black, Asian, etc.). Respondents with origins from the Middle East and North Africa (MENA) region also have difficulty finding a race category that appropriately represents their identity.
To address some of these issues, the new standards combine race and Hispanic ethnicity into one question and add a MENA race category. The new OMB guidelines only require that government agencies collect data from the new list of racial categories. The guidelines do not mandate how agencies collect those data, but they do provide examples of how agencies can list each racial category, including the use of examples such as specific Asian Indian in the Asian countries of origin.
Federal agencies must comply with the new requirements by March 2029, but it will be several years before standardization exists across agencies and data products, including the American Community Survey (ACS) and Current Population Survey (CPS). Many researchers and statisticians use the ACS and CPS as the basis of official estimates for the purposes of sampling and weighting. However, we currently know little about how these new required categorizations, and how the Census Bureau and the ACS will implement these recommendations, could affect the estimates of race and ethnicity.
Gallup’s Experiment
To address this need, Gallup conducted a large experiment in May 2024 with 46,933 panel members from the probability-based Gallup Panel via a web survey. The goal was to examine how different presentations of race categories, the merging of Hispanic ethnicity into a single item on race, and question wording affected the distribution of responses and compare each version’s results.
Because of the sample design and differential response rates across groups, certain groups were overrepresented in our analysis, including non-Hispanic White respondents. The results of this experiment were unweighted and only intended to compare question versions, not to create representative population estimates for race and ethnicity.
Respondents were randomly assigned to answer one of the three versions of the race/ethnicity questions, and the questions appeared at the end of a survey that covered a variety of topics. All respondents were asked the same questions, except for the experiment. We excluded respondents who did not provide a valid response (n=444) to the question(s) from our analysis (0.9% of respondents). This included 92 respondents from version 1 (0.6%), 178 from version 2 (1.1%) and 174 from version 3 (1.1%).
In versions 2 and 3, if respondents selected none of the responses (in other words, they left the question blank), we excluded them from the analyses. In version 1, we excluded respondents if they left the Hispanic ethnicity question blank AND left the race question blank. We also excluded them if they answered “no” to the Hispanic question and left the race question blank (because we could not code them into a valid race/ethnicity category).
In version 1, Gallup followed the old OMB standard, with Hispanic ethnicity asked first as a yes/no question. On the same survey screen, respondents were asked to specify their race, which included six categories; allowed respondents to select as many as apply.
In this survey and past Gallup surveys, we included “other” as an option for respondents who may be confused about the categories or feel that none apply, which is not one of the required OMB options. We kept the other option for this experiment to examine how Gallup Panel members answered the race and ethnicity questions in the past and how they answered them in this experiment, which we will share in a future analysis.
Version 1
Q1 Are you, yourself, of Hispanic origin descent, such as Mexican, Puerto Rican, Cuban, or other Spanish background?
- Yes
- No
Q2 Which of the following best describes your race? Select all that apply
- White
- Black or African American
- Asian
- Alaska Native
- Native Hawaiian or Pacific Islander
- Other
In version 2, Gallup used the new OMB standard for asking race and ethnicity, which combined race and ethnicity into a single question and added MENA as a category. We did not add an “other” category to this version because it is not part of the OMB’s new minimum standards. Further, we believe the new version of the question may address some of the shortcomings of version 1, and an “other” category may not be necessary. It should also be noted that the order of the response options is different from version 1, with responses appearing in alphabetical order.
Version 2
Q1 What is your race and/or ethnicity? Select all that apply
- American Indian or Alaska Native
- Asian
- Black or African American
- Hispanic or Latino
- Middle Eastern or North African
- Native Hawaiian or Pacific Islander
- White
In version 3, Gallup used the new OMB standards. This version followed version 2 but added OMB examples to each category. We used the exact questions and examples provided in the OMB standards.
Version 3
Q1 What is your race and/or ethnicity? Select all that apply
- American Indian or Alaska Native (for example, Navajo Nation, Blackfeet Tribe of the Blackfeet Indian Reservation of Montana, Native Village of Barrow Inupiat Traditional Government, Nome Eskimo Community, Aztec, Maya, etc.)
- Asian (for example, Chinese, Asian Indian, Filipino, Vietnamese, Korean, Japanese, etc.)
- Black or African American (for example, African American, Jamaican, Haitian, Nigerian, Ethiopian, Somali, etc.)
- Hispanic or Latino (for example, Mexican, Puerto Rican, Salvadoran, Cuban, Dominican, Guatemalan, etc.)
- Middle Eastern or North African (for example, Lebanese, Iranian, Egyptian, Syrian, Iraqi, Israeli, etc.)
- Native Hawaiian or Pacific Islander (for example, Native Hawaiian, Samoan, Chamorro, Tongan, Fijian, Marshallese, etc.)
- White (for example, English, German, Irish, Italian, Polish, Scottish, etc.)
Results
We compared the results of each question to understand how each version affects the estimates. Using the new questions, OMB outlines possible analytical approaches when presenting race and ethnicity data. This article shares our results using two of OMB’s suggested approaches. The results use the exact OMB categories and definitions.
The first approach was to show the percentage of respondents who selected each response option. Because respondents can select multiple response options, the categories are not mutually exclusive, and the frequencies sum to greater than 100%. As Table 1 shows, there are few differences between versions, except for the “other” category, which respondents did not see in versions 2 and 3, and the MENA category, which respondents did not see in version 1.
Additionally, slightly more respondents indicated they are Hispanic or Latino in question version 1 (7.4%) when it was asked as a separate yes/no question, compared with 6.9% and 6.8% in versions 2 and 3, respectively.
Overall, there are few differences between version 2 (no examples provided) and version 3 (examples provided), with the exception of the American Indian or Alaska Native category. More respondents selected this category in version 3, which showed specific examples.
Our second analysis grouped people into mutually exclusive categories that show the percentage of respondents who selected only one race/ethnicity. The final category combined respondents who selected more than one race or ethnicity. We grouped each respondent into just one category, and the frequencies of all categories sum to 100%.
The notable difference is, again, the Hispanic or Latino alone category. Respondents in versions 2 and 3 were more likely to be categorized as Hispanic or Latino alone (4.2% and 4.4% versus 0.1%). While in version 1, more Hispanic respondents selected Hispanic and White (as shown in the previous table), which we classified in the multiracial/multiethnic category.
Bottom Line
Gallup’s analysis shows that the approach to collecting race and ethnicity matters, and in particular will affect the estimates of Hispanic White and multiracial respondents. In our experiment, fewer respondents selected Hispanic and White when the questions were combined into a single question.
In the single-question format, more respondents were categorized as Hispanic alone, while in the two-question format, more Hispanic respondents were categorized as multiracial or multiethnic. Researchers should keep this in mind when analyzing or weighting data by racial and ethnic groups or creating their own categories. Our findings also suggest that including examples in the question (version 3) versus not including examples (version 2), which are both acceptable under the new OMB standards, may produce differences in results.
These findings have important implications for how race and ethnicity are captured moving forward. When considering switching to the new question versions, Gallup recommends the following:
- If researchers use race and ethnicity in weighting, continue using the old two-part question until official federal statistics (such as ACS and CPS) are available using the new single question. These agencies have not announced official plans or timelines for adopting these questions or when estimates will be available using the new questions version.
- Many researchers want to create more expansive or inclusive categories, especially for respondents who may not see themselves reflected in the prior version of the question. To accomplish this goal and maintain consistency and comparability with current official statistics (such as ACS and CPS), researchers should consider asking new or expanded categories, such as MENA or expanded Asian categories, as a follow-up question.
- Researchers can use and categorize the race and ethnicity categories for analysis in many ways. The categories should be relevant to and correlated with the research question. Sometimes practical challenges such as sample size may also dictate the categories that can be used in analysis. Researchers cannot use a singular approach for every study.
- When collecting and analyzing data, it is also important to consider sample size and the group being studied. Certain categories may have very small sample sizes, and depending on the sample and population, unique combinations of categories could even unintentionally identify a respondent (that is, a deductive disclosure).
As part of this experiment, Gallup also compared the results to questions that Gallup Panel members answered in the past to better understand how self-reports of race and ethnicity may change based on how the questions are asked. Gallup will continue analyzing these findings from our experiment and report additional insights in a future article or Gallup Methodology Blog.
To stay up to date with the latest Gallup News insights and updates, follow us on Twitter.
