Online panels: quick, easy, cheap, but potentially dangerous…

The technology used to conduct effective and efficient market research is growing by leaps and bounds every year.  We have gone from punch cards to fMRI (functional Magnetic Resonance Imaging to conduct neuromarketing studies in the space of just a few decades.

New technologies are wonderful things.  They allow us to do more with less in many ways.  That said, with new technologies comes new problems.  One of these new problems has been actively discussed on the LinkedIn Group NextGen Market Research (NGMR).  If you are a market researcher and haven’t joined this group, you should.  You will learn a lot from the discussions on this site.

The most recent topic that has been discussed has to do with a really important issue:  data quality in online panels.   Online panels can be very useful tools in collecting data both quickly and inexpensively.  There is just one big problem that this data collection tool experiences: the “professional” respondent or as I like to call them “cheaters”.  These are folks that are on many panels and/or are on the same panel many times under different names.  These folks can have an adverse effect on your data and ultimately the actions you take based on your data because they aren’t interested in giving you thoughtful responses to your questions, they are only interested in completing as many surveys as they can to receive the monetary or gift incentives they receive for completing surveys.

There have been several articles written on the subject lately and here’s why:  China/India Farms.  Unlike normal cheaters, these are people in China or India or other country that work for companies that are actively defrauding survey and gaming companies. These are people who are not qualified for the surveys they are filling out; and they are filling out these surveys as fast as they can.  Oh how I long for the days of the simple telephone survey!

But don’t despair; there are ways to effectively weed out the cheaters.  So, if you are planning to conduct research using an online panel, here’s what I would recommend you ask your online panel vendor. First and foremost: what methods do you use to weed out cheaters?

Here are the detailed questions you should be asking:

  1. Do you use Machine ID (picks up the IP address of the respondent) to weed out cheaters?
  2. Do you use digital fingerprinting and personal identity verification with a third party?
  3. What test questions do you use to identify cheaters? 
    • Do you use both open and closed ended questions to test for cheaters?
  4. Do you have your survey programmed such that it does a goodness of fit calculation for each respondent?  Meaning do you have measures/questions in your survey that are evaluated against each other that enable you to find incongruities in your data (and eliminate them)?
  5. Do you use the amount of time the respondents take to answer each question to weed out cheaters?
  6. What percentage of your survey respondents fall out of the survey due to straight-lining (giving the same numerical answer for every question) and/or non-sense responses?
  7. What percentage of your survey respondents fall out of the survey because they fail the test questions?

Online panels can be a very quick and cost effective way to collect data.  And, there are certain populations of respondents that are very suited to this type of surveying.  So, you should consider using them when appropriate.  Just do yourself a favor and do your homework.  Not all panels are the same or even good.   Ask questions; exercise your right to be informed!

Happy marketing!


2 responses to this post.

  1. Sharon:

    Good information. I know feedback is important to establish an organization’s public image, but without reliable sampling tools the organization’s decision-making process is corrupted. You’re seven recommendations/questions should be used a standard test for validity in any sampling.




  2. Posted by sharonmarkovsky on May 13, 2010 at 10:47 am

    Paul, thanks for the feedback! 🙂

    Nowaways, people are in such a rush to get feedback and get it as cheap as possible that they might not be as rigorous as they need to be with respect to checking up on the quality of the data produced by their market research vendors.

    Thanks for your insightful comment!


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: