Gallup Begins Research on Simulated Responses

WASHINGTON, D.C. — Recent innovations in artificial intelligence are opening new avenues for producing and analyzing data. One emerging approach uses AI-generated agents to create “simulated” responses that are designed to simulate how individuals and populations might answer survey questions. With these advancements come questions about their accuracy, suitability, transparency and ethical use.

Gallup is exploring the potential of simulated responses through a partnership with Simile, an AI company founded by Stanford researchers who pioneered research on this method.¹ Gallup is independently validating the Simile method to evaluate where these AI-generated agents perform well in predicting people’s responses, where they fall short and how they compare to established methods. Our goal is to learn whether AI systems and emerging methods can help deepen, not replace, our understanding of how humans think and behave. We are applying the same standards of rigor and transparency that guide all of our methodological research.

Building AI-Generated Agents From Probability-Based Data

Gallup’s exploration of simulated responses starts with high-quality, probability-based data collection. We have been conducting in-depth interviews with members of our probability-based U.S. panel to capture detailed and nuanced attitudes, beliefs and experiences that cannot be fully assessed with closed-ended questions alone.

Agents are created based on the responses to these interviews as well as traditional survey questions. Agents are then prompted to use relevant information from the qualitative and quantitative human responses to predict how people might respond to a new question or stimulus. Answers to a question from an entire agent bank (a collection of agents specializing in a topic) approximate results from a random sample of people.

Advancing Innovation With Clear Standards

The value of any methodology depends on whether it is fit for a specific purpose. Gallup is actively evaluating where simulated approaches produce reliable results, where they introduce bias and where they simply do not work.

Our commitment to high-quality, probability-based research that directly surveys real people has not changed. That work remains the most appropriate methodology for producing official statistics and population-level estimates. For clients who rely on Gallup for nationally representative insights grounded in rigorous sampling, that standard is not going anywhere. Our work on simulated responses is not a departure from that commitment. It is built on top of it.

What does this mean in practice? Simulated responses will not be used to produce Gallup’s published population estimates, and they will not replace direct measurement of people in our tracking research. The goal is to explore where this methodology can extend our capabilities, not to substitute it for work that requires the rigor of probability-based sampling. We will always be transparent about our use of simulated responses and will never present them as human responses.

At the same time, we see value in studying how simulated approaches could support specific use cases, in carefully designed ways. Researchers have long used statistical models to predict attitudes and behaviors, such as voting, purchasing decisions or reactions to policy changes. Simulated responses represent an evolution of that modeling tradition, using richer data to produce more nuanced predictions.

We see potential in several areas. In research design, models based on existing data could simulate discussions that help identify hypotheses or refine questions before researchers test them with real respondents. Simulated responses can also offer cost-effective ways to pre-test questionnaire items and investigate novel concepts before committing to a full field study.

Simulated approaches may also expand what is practically researchable, enabling organizations to explore attitudinal questions at a scale or speed that would otherwise be prohibitive, to model responses among hard-to-reach populations, or to test how different groups might react to novel stimuli before investing in full-scale data collection. In short, agent banks may help researchers and practitioners anticipate likely outcomes and support decision-making.

We will only consider broader applications if the evidence supports them. Any expanded use will require demonstrated accuracy, a clear fit for purpose, and transparency about how the data were generated, including their limitations.

Early Stages of Agent Bank Development and Validation

Starting in fall 2025, approximately 1,000 Gallup Panel™ members completed in-depth interviews, which were used to create agents. Using these agents, we have started validation exercises to measure how well simulated responses can reproduce known patterns in the data at the respondent, survey and population levels. To carry out this validation work, we compare the estimates generated from simulated responses to those from probability-based estimates collected from Gallup Panel respondents.

Our early findings show that for general population estimates of the questions we tested, particularly those closely aligned with the topics covered in the original interviews, the distribution of simulated outputs was close enough to approximating human responses for us to warrant continued exploration. We will share the results of our testing and validation in future articles.

Building on insights from this pilot stage, Gallup and Simile are currently developing new agent banks that cover different topic areas. Validation work will focus on how well simulated responses predict people’s responses across several objectives:

• gauging the extent to which the accuracy of estimates declines as questions diverge from the topics covered in the original interviews
• assessing how much predictive power of approximating people’s responses is gained as more information is added to the agents
• determining how well agents replicate known patterns across demographic subgroups
• examining consistency of performance across different question formats and response scales
• measuring how quickly agent predictions deteriorate over time as the world changes and agents are no longer updated

Continuing the Conversation

We acknowledge that this technology has the potential to erode public trust. That is why we are gathering empirical evidence independently, assessing the accuracy of this methodology rigorously, and determining the circumstances under which simulated responses can be used. We take seriously the questions this work raises about quality and trust. Credibility is central to everything we do and underlies the reputation Gallup has built over 90 years. If validated and used responsibly, this technology has the potential to provide organizations with rapid, reliable and more powerful insights.

Transparency is foundational to trust and becomes even more crucial as new approaches emerge. We will share the results of our validation work, the effectiveness of the method and any limitations we encounter. When we publish findings, we will clearly describe what was done, how data were generated, and whether results come from real human responses or from simulated responses. The two are not interchangeable, and we will never present simulated results using language that implies direct human measurement. We encourage others in the field to adopt equally clear standards.

[Editor’s note: This article has been updated since its original publication.]

Stay up to date with the latest insights by following @Gallup on X and on Instagram.