Andrew Wiseman looks at the race to the bottom for online sample, and the implications for research.
Online sampling has quickly become the lifeblood for the majority of quantitative research projects in many markets across the globe. Whether using specialised online research panels, where pre-screened participants complete surveys for financial reward, or using river sampling through affiliate websites, online sampling has quickly become established as the norm. However, the uncomfortable truth is that online sample has a BIG image problem…
I read a report recently that suggested as much as thirty five percent of online sample had been identified as unreliable. Let’s look at that again: THIRTY. FIVE. PERCENT. There’s many reasons for these unreliable responses – from lazy panelists speeding through a survey with little care and attention, to the more worrying trend of survey farms and AI respondents.
On the one hand, research agencies are keen to talk about the quality of the data that they use for generating insights. At the same time, they’re as keen to get the price as low as possible and get the fieldwork done as quickly as possible. Now, I was once told you can’t have all three. You can have fast and cheap, good and fast or good and cheap – but something’s got to give somewhere. Just remember. THIRTY. FIVE. PERCENT.
We also talk to our clients about the difference between price and value. Now, imagine buying 6 apples for a pound. You’ll probably be pleased with the price you paid (less than 17p an apple is pretty good, right?). But then what if around a third of them were rotten? Is 4 apples for a pound good value? (It probably is in today’s world of food inflation, but you hopefully get the point). PRICE ≠ VALUE! Rather, value is paying a fair price for the quality of the product or service you’ve bought.
This got me thinking about Van Westendorp’s price sensitivity meter. Many of you will be familiar with this pricing technique – and I wonder how it would look at panel sample? For those less aware, the model asks research participants to define the following four prices:
- At what price would it be so low that you start to question this product’s quality?
- At what price do you think this product is starting to be a bargain?
- At what price does this product begin to seem expensive?
- At what price is this product too expensive?
I wonder where your typical sub £1.50 natrep cost per interview would sit here? Should it sit in that first bucket, where the quality is questionable? That 35% stat would certainly suggest so!
So what can we do? We know that panel sample providers are doing a lot to try and measure the quality of participants. At Mustard we’re also doing a LOT of new and different things to catch the fraudsters in flight (we’ll keep our ‘secret sauce’ to ourselves – who knows what kind of AI bots could be listening in?). But is the best approach to go back to that initial question: Do I want good quality data. If the answer’s yes, then perhaps we should be prepared to either wait longer for it, or pay more for it, to help to turn the tide in data quality.