Anthony Shephard-Williams explores one of the most common questions in research: how many people do we need to speak to? He explains why there is no magic sample size, why bigger is not always better, and how the right approach depends on the decision you need the research to support.
One of the questions we get asked surprisingly often in research is: “Is that sample size big enough?”.
How many respondents do you need for a market research survey? Sometimes the number is 30, sometimes 50, sometimes 500. And the slightly frustrating answer is usually the same: it depends.
There isn’t a magic number at which research suddenly becomes robust. A sample can be perfectly sensible for one objective and completely inadequate for another. The more useful question is what you actually need to do with the findings.
Sample size should follow the decision you need to make, rather than the other way around.
So, why 40?
Anyone who has worked in market research for a while will probably have heard 30 talked about as something of a minimum threshold for quantitative analysis.
There is some statistical logic behind where that number comes from. In introductory statistics, n=30 is commonly used as a rule of thumb for when a sample becomes large enough for statistical results to start behaving in a more predictable and reliable way.
But it is important to stress that 30 isn’t a magic number either. It isn’t an absolute minimum below which statistical analysis becomes impossible, nor does reaching 30 suddenly make every comparison robust. Different analyses have different requirements, and the distribution of the data, size of the differences you are looking for and number of subgroups all matter.
That is partly why I like the question “Is 40 enough?”. Forty feels like it should be safely above that familiar rule of thumb but depending on what you want to do with those 40 people, it may be plenty or nowhere near enough.
Bigger is better, but only up to a point
Statistically, larger samples usually give us greater precision. But the relationship isn’t linear. As sample size increases, the additional precision you gain from each extra respondent gets progressively smaller. That creates an important point about diminishing returns.
As a rough guide, if around half of your sample gave a particular answer, the true result could reasonably sit within the following range:
Note: These figures are intended as a simple illustration. The exact level of uncertainty will vary depending on the type of research and analysis being carried out.
The law of diminishing returns
A useful rule of thumb is that halving the margin of error requires roughly four times the sample size. Moving from 100 to 400 respondents makes a meaningful difference, but achieving the same improvement again means increasing from 1,000 to 4,000. That is why gains become increasingly expensive as samples grow.
The important bit is the pattern. Bigger samples give us more precise results, but each additional respondent buys us progressively less. Sometimes that extra precision is exactly what the research needs, but “as big as we can afford” isn’t necessarily the smartest way to set a sample size.
In other words, bigger samples bring greater confidence, but with diminishing returns.
But how many people exist in the first place?
Another important consideration is the size of the total population, or ‘universe’, you are trying to represent.
I’m sure most people in the research industry have tales of clients asking for the most niche audiences and thinking it will be straightforward. Think males, aged 30–40, living in London, with three dogs and a rabbit, who shop at Aldi, drive an electric car and have switched broadband provider in the last six months. At some point, you stop talking about “the market” and start wondering whether the entire eligible universe could fit around one meeting-room table.
Now imagine there are only 50 people in total who meet your criteria. Speaking to 40 of them is clearly very different from speaking to 40 people from a population of millions. In simple terms, 40 out of 50 tells you far more about that specific population than 40 out of 50 million.
When the total universe is small, the question can therefore become less about following standard sample-size rules and more about how much of that universe you can realistically reach. You also need to consider how you are going to reach them, but that is a whole other blog in itself!
This is especially relevant in B2B and specialist research, where the total pool of eligible people may be very limited. In those cases, context matters just as much as the headline sample size.
A healthy total sample can hide some very unhealthy subgroups
This is probably one of the biggest traps in research.
On a recent project for a national garage chain, we had a very healthy overall sample of almost 2,000 respondents, which we needed to support both MaxDiff and TURF analysis. However, once we started looking at specific audiences, the picture changed.
One important lapsed-customer subgroup had a base of just 42. That wasn’t large enough for the statistical analysis we needed to run. Those respondents could still contribute to the overall findings, but we had to steer clear of reporting on the lapsed audience in isolation.
It was a good reminder that a strong overall sample doesn’t automatically mean every subgroup within it is strong enough to analyse.
A study can begin with a perfectly healthy n=500. Then someone asks to compare customers and non-customers, younger and older people, six regions and three different usage groups. Before long, you can find yourself looking at individual cells of 20 or 30 people.
Sometimes 40 really is enough
All of this might make it sound as though small samples are inherently problematic. They aren’t.
Imagine we are testing a new online customer journey with 20 people and 14 struggle at exactly the same stage. Do we really need to test another 480 before doing something about it? Probably not.
That doesn’t mean 20 people give you a statistically robust answer, but they can give you an important early signal. Our Strategy Director, Andrew Wiseman has written before about why the ‘first 20’ matter. Read it HERE. Early analysis can highlight when something isn’t working, giving you the opportunity to investigate or adapt before hundreds more interviews are completed.
The purpose of that research isn’t necessarily to estimate precisely what percentage of the entire customer population will experience the issue. It is to identify whether a problem exists, understand why it happens and work out how to improve it.
The same applies to more intensive research. A trial involving 40 people who need to onboard, use a new service over several weeks, complete tasks and attend a follow-up interview is very different from 40 people answering a 10-minute online questionnaire.
Each participant is generating much richer evidence, and pushing that sample from 40 to 100 simply because “40 sounds small” may add significant cost and complexity without adding an equivalent amount of useful learning.
The useful distinction is often between understanding something and measuring something precisely.
Be careful how you present small samples
There is another issue with small samples that has less to do with the analysis and more to do with how we communicate it.
Compare:
“65% preferred Concept A.”
with:
“13 of the 20 participants preferred Concept A.”
Mathematically, they are saying the same thing. But they don’t feel the same.
The first can sound like a definitive statement about a population. The second immediately tells the reader how much evidence sits behind it.
That is why I always favour counts, clearly displayed base sizes and plain-English descriptions when samples are small. It isn’t about hiding or dismissing the findings. It is about presenting them with an appropriate level of confidence.
A quick sample-size sense check
There is no universal table that tells you the “right” sample for every project, but these are useful questions to ask before deciding:
Sample size isn’t the same thing as research quality
Perhaps the most important point is that a huge sample doesn’t rescue poor research design.
A survey of 5,000 poorly recruited respondents answering a confusing or leading question can still produce a misleading answer. A smaller study with the right participants, appropriate tasks, good questions and thoughtful analysis can provide very strong evidence for the decision it was designed to inform.
Personally, I’d much rather have 40 of the right people doing the right task than 400 people recruited largely because the bigger number looks reassuring on a methodology slide.
There is no universal “good” sample size.
40 can be plenty. 400 can sometimes be too few.
The job of a researcher isn’t simply to maximise the number of respondents. It is to design enough research to support the decision being made, and to be transparent about what the evidence can and cannot tell us.
Ultimately, a sample doesn’t need to be enormous to be useful. It just needs to be fit for purpose.
Four questions to ask before judging a sample size
Before deciding whether a sample is “big enough”, it helps to look beyond the headline number and ask four simple questions:
- What are we trying to learn or decide?
Exploring a problem, testing an experience or identifying issues can require a very different sample from accurately measuring how common something is. - How precise does the answer need to be?
If a small difference could change the decision, you’ll usually need a larger sample. If you’re looking for a clear directional steer, you may not. - How big is the total population we’re trying to represent?
Forty people from an audience of 50 is very different from 40 people from a population of millions. - Which groups do we need to analyse separately?
A healthy overall sample can quickly become much smaller once you split it by customer type, age, region or another important subgroup.
There isn’t one magic number that answers all four questions. The right sample is the one that gives you enough confidence for the decision you actually need to make.
Choosing the right research sample size depends on more than a simple number. Effective research design considers the objective, target audience, level of precision required, subgroup analysis and how the findings will be used. A smaller sample can provide valuable insight when the research is designed correctly, while a larger sample does not automatically guarantee better answers.
The right sample size is not the biggest number you can afford. It is the amount of evidence you need to make a confident decision.
At Mustard, we help organisations design research that is fit for purpose, balancing statistical confidence, practical constraints and the decisions the research needs to support. Whether you need to understand customers, test a new proposition or measure market opportunity, we make sure the methodology is designed around the insight you actually need.
If you’re planning research and want to understand what sample size you really need, get in touch.