The technology used to conduct effective and efficient market research is growing by leaps and bounds every year. We have gone from punch cards to fMRI (functional Magnetic Resonance Imaging to conduct neuromarketing studies in the space of just a few decades.
New technologies are wonderful things. They allow us to do more with less in many ways. That said, with new technologies comes new problems. One of these new problems has been actively discussed on the LinkedIn Group NextGen Market Research (NGMR). If you are a market researcher and haven’t joined this group, you should. You will learn a lot from the discussions on this site.
The most recent topic that has been discussed has to do with a really important issue: data quality in online panels. Online panels can be very useful tools in collecting data both quickly and inexpensively. There is just one big problem that this data collection tool experiences: the “professional” respondent or as I like to call them “cheaters”. These are folks that are on many panels and/or are on the same panel many times under different names. These folks can have an adverse effect on your data and ultimately the actions you take based on your data because they aren’t interested in giving you thoughtful responses to your questions, they are only interested in completing as many surveys as they can to receive the monetary or gift incentives they receive for completing surveys.
There have been several articles written on the subject lately and here’s why: China/India Farms. Unlike normal cheaters, these are people in China or India or other country that work for companies that are actively defrauding survey and gaming companies. These are people who are not qualified for the surveys they are filling out; and they are filling out these surveys as fast as they can. Oh how I long for the days of the simple telephone survey!
But don’t despair; there are ways to effectively weed out the cheaters. So, if you are planning to conduct research using an online panel, here’s what I would recommend you ask your online panel vendor. First and foremost: what methods do you use to weed out cheaters?
Here are the detailed questions you should be asking:
- Do you use Machine ID (picks up the IP address of the respondent) to weed out cheaters?
- Do you use digital fingerprinting and personal identity verification with a third party?
- What test questions do you use to identify cheaters?
- Do you use both open and closed ended questions to test for cheaters?
- Do you have your survey programmed such that it does a goodness of fit calculation for each respondent? Meaning do you have measures/questions in your survey that are evaluated against each other that enable you to find incongruities in your data (and eliminate them)?
- Do you use the amount of time the respondents take to answer each question to weed out cheaters?
- What percentage of your survey respondents fall out of the survey due to straight-lining (giving the same numerical answer for every question) and/or non-sense responses?
- What percentage of your survey respondents fall out of the survey because they fail the test questions?
Online panels can be a very quick and cost effective way to collect data. And, there are certain populations of respondents that are very suited to this type of surveying. So, you should consider using them when appropriate. Just do yourself a favor and do your homework. Not all panels are the same or even good. Ask questions; exercise your right to be informed!