Synthetic data promises faster, cheaper research, but how far can we trust it? Anthony Shephard-Williams explores where synthetic data could add value to market research, where the risks become harder to ignore, and why the consequences of getting an answer wrong should determine how much we rely on it.
Over the past couple of years, I’ve been to a lot of research and AI industry events where synthetic data has been discussed. In market research, synthetic data is increasingly being presented as a way to simulate respondents and generate research findings without recruiting the people themselves. The conversation nearly always starts with the opportunity: faster research, lower costs, fewer recruitment challenges and the ability to test more ideas more quickly.
Then come the caveats. Bias, stereotyping, lack of variation, difficulties with niche audiences, problems with genuinely new behaviours and the challenge of recreating the messy, contradictory way that real people actually think and behave.
By the end of many of these sessions, I’m often left thinking that there are an awful lot of watchouts attached to something we’re increasingly being encouraged to treat as research data.
Disclaimer. We haven’t used synthetic data at Mustard. Yet.
That doesn’t mean I’m against it. We haven’t used synthetic data on a Mustard project yet, but we’re watching the market closely, looking at the experiments taking place and keeping an open mind. If we do use it, it will be because it is right for that project and because we have enough confidence in what the resulting data can genuinely tell us.
Plausible isn’t necessarily true
Large language models are incredibly good at producing answers that sound like something a real person might say. The issue is that plausible and accurate are not necessarily the same thing.
Real people are inconsistent. We say price matters and then buy the expensive option. We are influenced by habit, emotion and context, and often struggle to explain our own behaviour afterwards.
Recent research highlights that challenge. In 2026, researchers working with Prolific tried to recreate the responses of 996 real US participants and found that richer demographic personas did not automatically improve synthetic results. Their conclusion was that synthetic respondents currently look better suited to directional and exploratory work than replacing human measurement. [1]
STRAT7 reached a similarly mixed conclusion in research involving around 3,000 UK respondents. Synthetic results could get reasonably close on some headline measures, but differences became much more apparent when looking at subgroups, segmentation, drivers and changes over time.[2]
For me, all of this suggests that the useful question isn’t simply “does synthetic data work?”
It is “what does it work well enough for?”
Where could synthetic data help in research?
I can see real potential at the exploratory end of research. Synthetic respondents could help generate hypotheses, challenge assumptions, stress-test questionnaires or propositions and identify issues worth exploring before spending money on fieldwork.
I can also see a role in prioritisation. If AI helps generate 100 ideas, synthetic approaches might help reduce that to the 20 worth investigating properly.
Where I’m much more cautious is measurement. If we need to know whether 41% of customers believe something, whether satisfaction has genuinely fallen, or whether one customer group behaves differently from another, I still want that evidence to come from the people we are actually trying to understand.
A useful test might simply be:
What happens if this answer is wrong?
The bigger the consequence of getting it wrong, the stronger the case for validating the answer with real people.
What should you ask before using synthetic data?
Ask what sits underneath it.
We need to be careful about treating synthetic data as one methodology. A general-purpose LLM pretending to be your customer is very different from a model grounded in years of first-party research and repeatedly validated against human samples.
If (and remember, that’s still a big IF) I was to consider synthetic data for market research, I’d want answers to three questions:
- What real data is it grounded in?
- How has it been validated against real people for this particular task?
- Where has it been shown not to work particularly well?
Why #REAL still matters
One of our three brand values at Mustard is #Real.
For me, that doesn’t mean resisting new technology. It means being realistic about where new tools genuinely improve research and where they may introduce more uncertainty than they remove.
I fully expect synthetic data to become part of the research toolkit. We’ll keep watching, learning and testing the evidence as it develops.
But our other values – #Savvy and #Confident – are equally applicable to this debate. We are savvy enough to use the shiny new tools and techniques when we can be confident that it’s the right choice.
For now, and for when the decision really matters, I still think there is something reassuringly valuable about hearing from the people themselves.
Considering synthetic data for your research? At Mustard, we help brands work out where AI and new research approaches can genuinely add value, and where hearing from real people remains essential. If you’re exploring synthetic data or looking at how to make your research faster and more effective, get in touch.
Sources
[1] Gordon, A., Petrova, N., Burden, J., Bosch, O. & Ding, N. (2026). Do Richer Personas Improve LLM Survey Simulation? A Fidelity Paradox. Prolific / SSRN.
[2] STRAT7 (2026). Synthetic data: Is this as good as it gets?
[3] Muthukrishna, M. & Warner, B. (2026). Where You Ask Matters: Platform Effects of Online Surveys and AI-Based Human Subject Simulation. LSE / SSRN. The paper discloses funding from Electric Twin and links between the authors and the company.