Measuring Diverse Gender Identities: Phase 2

WASHINGTON, D.C. -- Understanding and giving a voice to the diverse experiences that people have is an important part of social and demographic research. Historically, however, some segments of the population -- such as gender minorities -- have not been measured and given that voice. Although researchers and federal statistical systems are now working to overcome the data gap for gender minorities, more work is needed.

One challenge for gender research involves creating questions that accurately measure diverse gender identities. To overcome this challenge and accurately measure the experiences of all survey participants, Gallup conducted extensive testing of new, more inclusive and accurate gender questions.

Sometimes a single question capturing broad -- but still inclusive -- gender identity suffices. Other times, research may require more nuanced gender identity reporting. In the first article, we discussed how we developed a baseline gender measure that is a single, simple question and includes an option other than male or female. This article focuses on the development of more detailed items that can be applied in a variety of research settings.

We tested four different question versions in a web survey experiment of 22,882 U.S. adults, conducted April 3-13, 2023, via the Gallup Panel. Respondents were randomly assigned to answer one of the four question variants.

It was also important to us to include the input of non-cisgender experiences in this research. After the gender question(s), respondents who selected an identity other than male (man) or female (woman) were asked to provide open-ended feedback on the questions they were asked (see footnote for exact wording).¹

Question Versions

Version 1

In the first version, we asked the single, broad gender question that is fielded on many of our U.S. surveys of the general population. This is the version of the question we adopted after our first round of experiments.

What is your gender?

1. Male
2. Female
3. Nonbinary

Version 2

In the second version, we modified the first question to adopt gender terminology (man and woman) versus terms used to describe sex (male and female). A common critique of gender questions, including our version 1 question, is that they ask about gender but use terms for sex in the response options. We wanted to test whether a change in terms would produce a difference in our estimates.

What is your gender?

1. Man
2. Woman
3. Nonbinary

Version 3

The third question we tested is the wording recommended by the National Academies in a 2022 Issue Brief on measuring sex, gender identity and sexual orientation. Questions like this are important not only because they present respondents with options that reflect their identity but also provide categories that allow researchers to measure and understand the diversity of gender experiences.

What sex were you assigned at birth, on your original birth certificate?

1. Male
2. Female

What is your current gender?

1. Male
2. Female
3. Transgender
4. Two-spirit
5. I use a different term (free text)

Version 4

In the fourth version, we modified the National Academies’ wording in five important ways:

1. We switched the gender question to use gender terms (man and woman).
2. We allowed people to select more than one option, which recognizes that gender is complex and does not always fit into one prescribed category.
3. We changed the transgender category to “trans or transgender,” as “trans” is a commonly used term.
4. Next, we eliminated the option for two-spirit. Our decision to do so was not to minimize the experiences of Native American populations but rather to acknowledge that there are many culturally specific terms (for example, “mahu” in Native Hawaiian and Tahitian cultures) and that listing only one may in and of itself be less inclusive. Any of these terms can be shared in the “I use a different term” option.
5. We added the category “nonbinary.” In our previous research, we found respondents most commonly used this term to describe a gender identity other than male or female.

What sex were you assigned at birth, on your original birth certificate?

1. Male
2. Female

Which of the following best describes your gender? (You may select all that apply.)

1. Man
2. Woman
3. Trans or transgender
4. Nonbinary
5. I use a different term (free text)

Results

When comparing the first and second versions, we found no significant differences in the results. Switching from terms used to describe sex (male and female) to gender (man and woman) did not produce a difference in our estimates.

Next, we compared the results of versions 3 and 4, which were two-part questions asking about sex and gender. Both questions started with a question asking about sex assigned at birth. Both question versions were the same, and as expected, the results of this question were nearly identical for versions 3 and 4.

We then compared the results of the gender questions. Version 3 used terms to describe sex, while version 4 used terms to describe gender. As we saw when comparing versions 1 and 2, we observed no meaningful difference between groups 3 and 4 in the percentage of respondents who reported male/man or female/woman.

We changed version 4 to adopt a “select all that apply” format. This change did not produce any significant difference in the distributions of male/man and female/woman. In version 4, 0.8% of respondents (n=47) selected two options and 0.1% (n=4) selected three options.

In version 3, 0.1% of respondents (n=6) selected the two-spirit option. Version 4 displayed this option to all respondents, regardless of reported race or ethnicity. Of the six respondents who selected the two-spirit option, one identified as Native American.

In version 4, 1.0% of respondents (n=57) selected the nonbinary response option, which was the primary difference observed between version 3 and version 4.

Respondent Opinions of the Question Versions

After each question version, we asked all respondents who selected any option(s) other than male/man or female/woman to provide their feedback on the questions. Of the 233 people who were asked the open-ended question, 116 provided a response. An analysis of the qualitative feedback identified several themes:

• In versions 1 and 2, which presented only three response options (male, female, nonbinary; man, woman, nonbinary), 26 of the 37 respondents who wrote an answer expressed a desire to have more categories or a box where they could fill in an answer.
• In versions 3 and 4, which presented two questions that asked about sex assigned at birth and current gender, about a third of the qualitative responses (n=27) were from people who expressed their dissatisfaction that the gender questions included more than two categories. We read feedback such as “There are only two genders” and “You are born XX or XY; you don’t get to pick.”
  
  Although we tried to limit the open-ended response to non-cisgender respondents, these respondents selected “I use a different term” in the quantitative question. This option contained an open-ended text box and was used to write complaints about the nature of the question. It is important to note that the group that dishonestly answered the initial gender question and provided critical feedback accounted for 27 out of 22,882 responses. This means that 99.9% of all survey respondents answered the initial gender questions without incident.
• Respondents in version 3 expressed frustration that they were only allowed to select one option. We heard comments such as the following:

“Gender identity is a continuum, and some of your questions offer response choices that are binary and deterministic.”

“‘Transgender’” feels like it precludes me from male or female.”

“Transgender is not a gender in and of itself and should not be listed in contrast with male or female….”

• Several transgender respondents expressed concern with being asked about their sex assigned at birth. We received feedback from transgender respondents who felt this question should not be asked unless relevant to the specific research question. One transgender respondent also indicated this question could be triggering for trans people because it asks them to deny their gender.

“Sex at birth is not relevant to anything other than medical questions.”

“Assigned sex at birth is not an acceptable question in any circumstance, other than maybe certain medical situations.”

Conclusions and Recommendations

From this research, we developed several recommendations. These can be taken into consideration specific to the research objectives and target population of a given study.

• Use gender terminology (man and woman), not sex terminology (male and female).
• Allow respondents to select all gender categories that apply.
• Do not ask about sex assigned at birth unless it is relevant to the research objective.
• Include a nonbinary category. This option can also help reduce the number of write-in responses and coding that is necessary.
• If using the term “two-spirit,” keep in mind that some survey designs (such as paper and pencil) cannot be designed to only show this option if Native American is selected in a race or ethnicity question. Listing only two-spirit but no other culture-specific terms could be considered less inclusive.
• Consider adding “I use a different term” when a shortened set of response options is presented (such as versions 1 and 2). This option can be included with or without a text box. A text box may allow the researcher to capture responses not reflected in the answer choices, but coding these responses can add time.
• Some respondents will use write-in options as an opportunity to provide their personal thoughts on gender. Some of these comments may be highly offensive and triggering to some individuals. This should be taken into account when considering who should code open-ended responses and when determining how/if responses should be cleaned before being analyzed.

Our research included questions about transgender identities, but our results were not conclusive and are not discussed in detail in this article. This component of our research will be shared in a future blog. However, a clear finding is that trans is not in and of itself a gender identity and should not be treated as such in gender questions.

Finally, we want to acknowledge that accurately measuring gender in a survey question is complex. Identities and experiences are diverse, gender is not always binary, and terminology continues to evolve. Our recommendations are intended to be just that -- recommendations. There is currently no perfect question that fits all research objectives, but the pursuit of a perfect question shouldn’t prevent researchers from asking about gender using the current understanding of best practices. We hope this research adds to the knowledge and discussion about gender survey questions and that research in this area continues to develop and refine gender questions.

To stay up to date with the latest Gallup News insights and updates, follow us on Twitter.