In his latest blog, Steven Searle (data operations manager) lifts the lid on his time signed up to one of the leading online market research panels.
I have a confession to make. I’ve been undercover with panel companies for a number of years now. Previously this was a tactic of mine for ‘borrowing’ concepts for any new and exciting ways in which questions can be asked to participants. More recently, though, the focus of my attention has been to see what companies are/aren’t doing to combat survey fraud.
What have I found?
Don’t get me wrong, some stuff is happening to battle survey fraud (wahoo!) although there are some fundamental flaws that are quite easy to spot if you know what you’re looking for.
Due to the fact that I can’t be arsed running this past HR before publishing I’ll refer to the panel that I’m signed up to as Panel 1.
(If you’re curious: they’re a biggun)
Problem 1: Inconsistent use of quality measures
There are shed loads of surveys available through various different partner sources. What they don’t tell you is that the quality measures they send over to you (in a very professional looking pdf document) are basically redundant if they end up using panel partners and, in some cases, only actually apply to their “full service offering” (basically when they script and host the survey for you themselves).
Just for argument’s sake, let’s say that I explicitly wanted to use < Insert name of expensive panel > because their quality is known for being the best in town but they were struggling for numbers. What do they do?
Well they just throw in a quick API to the nasty, dirt cheap panels you’ve been looking to avoid. There are two things wrong with this:
- They don’t tell you – it just looks like they’ve upped their game significantly and you’re not one to argue because you’ve got a project to deliver
- When bringing a partner on board they’re relying on the pre-screening, including quality procedures, to come from THE PARTNER’S side rather than conducting their own independent verification methods. If they’re the cheap and nasty ones, this is not good!
Using Panel 1 as an example, on roughly 20% of the surveys I accessed when undercover, they use some external quality verification platform which, from looking at their website, is pretty good and stretches beyond the old digital fingerprinting stuff you usually see. It has a short MaxDiff-esque exercise to run through each time. Looks great so well done Panel 1… but it means sod all when it’s not applied to 80% of projects.
Problem 2: Incest
Who’d have thought that online data collection would be comparable to Game of Thrones.
Jokingly, at least in the UK, it’s mentioned about how incestuous the Market Research industry is and panels are no different.
As a survey programmer by trade, I can spot which panels are being used as partners because I know all of their link domains that are used to direct members from the screening section to the actual survey commissioned by the end client. This is done by redirecting between different websites. Anyway, I recently noticed something strange with one of the surveys that I wanted to take part in:
- Initial Screening: It started with a link asking a few demographic questions from Panel 1 – as expected
- Additional Screening: the link was then redirected to ask a couple of demographic questions on a link from the 1st low-quality/bargain basement panel (that I wouldn’t go near)
- More Screening: the link then redirected to ask a couple more demographic questions on a new link from the 2nd low-quality/bargain basement panel (again, another one that I wouldn’t touch with a bargepole)
- Start Survey: Finally, it circled back to the original Panel 1 link to begin the actual survey which, strangely, was being conducted by them the whole time
So, long story short, Panel 1 outsourced their own job to…well… themselves via two different partners! Wtf!?
As a more general thing, because the above was just an extreme example, I’ve seen mainstream panel partners in cahoots with each other on around 50% of the surveys I’ve tried to fill out.
Problem 3: multiple devices/accounts
I thought why not try the old ‘switch to mobile and use a different email address’ trick and see how things fared.
Now, as a tech wizard myself, I’ve got a bit of sympathy here. Not to bore you, but using a different device gives you a different IP address. Then if you also use a different email then you’re in until someone manually tracks you down which’ll never happen.
I mean, Netflix are successfully able to do this so why aren’t panels?
A few reasons and, again, sorry I’m going to have to bore you a little.
To successfully remove this completely what you would need is explicit consent of the user’s location for everyone on your panel. Static IP addresses are becoming a thing of the past due to there simply not being enough IP addresses in the world. When using Dynamic IP addresses it becomes harder to pinpoint location so you also need device information to properly triangulate the location.
Panel retention being as poor as it is means that no company in their right mind really wants to have to go through the process of putting off people from getting their beloved 30p for a 10 minute survey. Even if they did it’s not a silver bullet because I haven’t even broached the concept that maybe, just maybe some people don’t live in a house by themselves!
What are we doing about this?
The key takeaway from everything that we know about data quality is that the burden of responsibility must lie with us at Mustard and NOT our suppliers.
We know three important things:
- Who the rubbish suppliers are
- The need to adapt and create new quality control measures is constant, not just a ‘pet project’ for a month or two
- That the best point to catch out whether someone is fraudulently answering a survey is in the survey itself
As recently as the last 3 months we’ve rolled out 4 new quality control metrics that we use on EVERY panel data collection project. We have SLAs in place and don’t even bother with the cheap, rubbish aggregators of this world who are just glorified API machines that add no value to our lives whatsoever. And, lastly, we also understand the importance not just of the automated routes but looking at the data collectively and asking the question, “do I believe this?”.
Despite this, we’re not stopping at that. We’re continuously looking at ways to improve panel quality. If you want to find out more, feel free to get in touch via theteam@mustard-research.com