Andrew Wiseman explores the oft-discussed issue of whether benchmarks are genuinely useful. Or most likely not.
Yesterday I was in a pitch meeting for a global Customer Experience tracker.
The conversation was good. We talked about customer engagement, approach, storytelling, and how we’d turn insight into action. The client seemed engaged.
Then came the question that made us pause. Benchmarks. It’s the question every CX and NPS tracker eventually gets asked: how do you know if your number is good? Or more specifically: how do we compare to everyone else?
On the surface, it’s a fair question. If you’re investing in a CX programme, you want some sense of whether your score is strong, weak or somewhere in the middle. But the more I think about it, the more I think benchmarks are often one of the least useful parts of the conversation.
Sometimes they’re mildly interesting. Sometimes they provide useful context. And sometimes, to put it bluntly, they’re bollocks.
Here’s why.
1. Most benchmarks compare things that were never comparable
Brands like the comfort of an external number. It creates the impression of objectivity. The problem is that the comparison often makes very little sense.
Different categories create different expectations. The experience customers expect from a technology brand is very different from the experience they expect from an insurance provider or a utilities company.
Yet CX benchmark charts often mix categories freely.
- Why would an insurance brand want to compare its NPS to Apple?
- Why would a telecoms provider care whether it sits above or below a premium airline?
The comparison might look interesting on a slide, but it rarely tells you anything useful about the experience you deliver.
This becomes even more problematic in global work. Cultural differences shape the way people answer surveys. Some markets are more generous with scores. Some are more moderate. Some are more critical.
A “good” score in one country may represent something quite different in another. When those differences are ignored, benchmarks stop providing context and start creating noise.
2. If the data wasn’t collected in the same way, the comparison is already shaky
Even when the category looks sensible, another issue appears quickly.
The research itself was likely not conducted in the same way. For a benchmark to be genuinely robust, the underlying data needs to have been captured under similar conditions.
- The same sample frame.
- The same survey structure.
- The same question wording.
- The same timing.
- Ideally the same research mode.
In reality, that almost never happens.
One study might be conducted online while another is carried out by telephone. One survey might ask the NPS question at the beginning, another at the end. One tracker might survey very recent customers, while another includes anyone who has interacted with the brand in the past year.
Small differences in methodology can move scores more than people expect. I remember a client testing a move of B2B NPS from telephone to online. The difference: a negative 25 point shift in their score.
At that point the comparison becomes less about customer experience and more about survey design.
Large benchmark databases often look authoritative because of their scale. They include lots of brands and lots of data points. But unless that data has been captured in a consistent way, the comparison is weaker than it appears.
A benchmark is only as good as the comparability behind it.
3. Obsessing over competitors can distract you from fixing your own problems
There is also a more strategic issue with benchmark thinking. When organisations focus too heavily on their position relative to competitors, they can lose sight of what the research is actually supposed to do.
CX measurement should help you understand the experience customers are having with your brand.
- Where are the pain points?
- Which touchpoints create advocacy?
- Where does the journey break down?
Those questions lead to improvement.
Knowing that a competitor sits three points higher on NPS might create urgency, but it rarely explains what needs to change.
There is also a creative risk. If everyone starts chasing the same benchmarks, everyone starts chasing the same solutions. The same service features appear everywhere. The same improvements appear in every digital journey.
Before long the experience becomes vanilla. Competitive advantage rarely comes from copying the market. It comes from understanding your own customers better than other brands understand theirs.
4. This isn’t just a CX problem. Brand tracking has the same issue.
Benchmark thinking has been part of brand tracking for years.
- “How does our awareness compare to the category?’
- “Is our consideration score strong?”
- “Are we ahead of competitor X?”
Again, these questions sound reasonable. But they often suffer from the same issues as CX benchmarks.
- Different trackers define the category differently.
- Different surveys prompt different brand lists.
- Different sample frames produce different baselines.
One study might measure awareness among category buyers. Another might measure it among the general population.
One tracker might include five brands in the competitive set. Another might include twelve.
When those conditions change, the benchmark changes with them. What looks like a stable comparison can actually be the result of different research designs.
Without understanding those differences, the numbers can give a false sense of certainty.
5. What matters more than a benchmark is relativity within your own system
External benchmarks can provide a little context, but the most useful context often comes from the way your own research programme is designed.
A well-constructed tracker allows you to interpret results without relying on questionable external comparisons.
- You can compare performance across touchpoints in the journey.
- You can identify the factors that drive advocacy and satisfaction.
- You can analyse different customer segments.
- You can track change over time.
Some programmes also include simple measures of perceived differentiation. Customers can tell you whether the experience feels better, worse or about the same as other brands they deal with.
That kind of relativity is grounded in your own customers and your own experience.
It provides insight you can act on.
Benchmarks: the uncomfortable truth
Benchmarks are appealing because they simplify complexity. They give stakeholders a clear number and a quick reference point.
But customer experience is messy. Human behaviour is messy. Research data is rarely as tidy as we would like it to be.
Used carefully, benchmarks can offer a rough sense of context. Used without enough scrutiny, they create false certainty.
If I had to choose between a tracker with no benchmark but strong diagnostic insight, and a tracker with a glossy benchmark and very little explanation, I would take the first one every time.
Because the purpose of CX measurement is not to admire a number. It is to understand what customers are experiencing and what needs to improve.
In short, the point of research is not where you rank it’s knowing what to do next.