Lead from Knowledge: Interpreting Employee and Customer Research

NOTE: This is a revised, republished article.

Pundits and pollsters use marketing and social research to investigate opinions, make policy and design strategy.  Politicians use them as convenient information when the data support their points of view.

Business and political leaders should be familiar with the tools and techniques of marketing research for three big reasons:

1. It tells you what customers want and need.  That's the reason you have a business.
2. It helps you understand whether your employees are engaged or not.  They have to serve customers and carry your flag.  Don't forget them.
3. Finally, it's deep insight into what your competitors are up to.  You don't always have to react to the competition (some would argue you shouldn't), but can spot holes in their strategy when you talk to their customers.
   

As a marketing researcher for 20+ years, I spend a great deal of time helping people design studies, field and interpret them to better understand their business and other objectives. 

Figures lie and liars figure, right?

Not always.  Being skeptical of the methods of marketing research is healthy.  Those conducting these investigations often use inappropriate methodologies, poor sampling, bad questionnaire design and atrocious analyses.  But being a critical consumer of marketing research will help you not only be smarter in strategy and debates about the results, but also a better leader as well.

Let's start with sampling methods. 

Marketing research studies were originally conducted face-to-face, door-to-door at people's homes.  Social scientists used Census data to figure out how many people were needed from each Census tract to make a representative sample.  But, with the rise of crime in many areas, pollsters walking door to door were presented with many, um, challenges, in recent years.

Widespread telephone penetration in the middle of the last century led to the increased use of telephone surveys.  Most people had a telephone, and using AT&T's complex exchange assignment algorithms, pollsters were then able to calculate who lived in which areas, and essentially replicate what Census block walkers used to do with door to door methods.  Even de-listing your phone number won't avoid having you being called by sophisticated research firms, since most were able to randomly generate telephone numbers using the exchange information they received from Ma Bell and her little Bellettes.

Needless to say, abusive direct marketing practices have made telephone surveying challenging (despite the fact that politicians have cleverly left polling out of most legislation designed to protect people from unwanted interruptions at home), but this is still the most effective way to reach consumers in most countries today.

The increasing penetration of home computers and internet usage has created opportunities to survey people in panels, designed by social scientists to be nationally representative.  Online marketing research has increased in popularity as well, but today, is still somewhat less representative than telephone surveys.  Consider that many groups are still very underserved with online access (African Americans, unaculturated Hispanics, etc.), and others are slow to adopt online practices (e.g., seniors, less well educated, etc.).

These differences are decreasing, however, as online usage becomes more pervasive nationally.

Other surveying methods like mail, and combination methods like postcards directing voters to an online survey or an automated telephone survey suffer from low response rates (making them expensive), and are also associated with a source of survey error called "non-response bias."  All survey methods suffer from this, and the only solution a researcher has is to have good information upfront about what a representative sample would look like, and endeavor to fill quotas to ensure that the study remains representative.

When you first look at marketing research report, look for the sampling method.  Was it a telephone survey, online study or mail method?  If it was telephone, know that with over 100 million telephone equipped homes in the US, a reputable researcher had lots of possible people to sample from.  If it was an online or internet study, be slightly more suspicious.  There are no national directories of email addresses yet, so there is no way to sample randomly from among all online users.  AOL may know who all of their members are, but that leaves out all non-AOL users.  Online panels are often recruited by other means (calling or email-blasting people), and reputable researchers will often have access to panels that are designed to be nationally representative. 

However, it is still likely that certain groups will be under-represented in these studies, like the seniors or non-English speaking Hispanic population.  Even stacking the panel full of sought-out seniors or Hispanics poses a methodological problem:  if most people in these groups are not online, those who are, are innately different than everyone else.  They are more sophisticated technologically, usually better educated or have higher incomes.  Ask yourself whether you believe that these people really represent the people they are supposed to.

Survey design, including writing a decent questionnaire, is the one area of research that can be suspect.  When reviewing a survey, if the question isn't listed, wonder why.  How are you supposed to interpret the answers if the question isn't known?  Consider the veracity of the questionnaire writer

A recent poll I participated in had the following question: "How much would you say you support the President's policies in the Middle East?  Would you say that you support them fully, support them somewhat, or are you a whining liberal Communist, trying to wreak destruction and death on all true Americans?"  I'm exaggerating, of course, but the poll was written to obtain the pollster's point of view, not learn how I feel about the situation.

It's also important to look at the types of answers given to a question.  In a rating question, like my silly example above, one of the answers is often "Don't Know/Not sure." While that may be appropriate, whether or not the researcher leaves in the people who answer this is sometimes not clear.  Leaving in the Don't Know answers is completist, but it sometimes deflates the attitude ratings you're looking for.  Many researchers will report the percentage or number of people who don't have an opinion, and then report how those who DID have an opinion answered.  That's a better way to handle that.

Finally, there is the often reported margin of error associated with research surveys.  A "margin of error" is a statistical way of estimating the stability of the results.  This number comes from a formula based upon the number of people who participated in the survey, sometimes the size of the population (more important for small universes, like Communists living in Wisconsin, or PETA members in Manhattan), the actual percentages found in a survey question.

Many polls simply report the MAXIMUM error associated with the sample size of the survey.  This is often reported as +/- 5%.  First, know that sampling error (the real name of this statistic, by the way) is actually reported in PERCENTAGE POINTS, not percent.  So, the right way to give the sampling error in my example is +/- 5 percentage points.  Sampling errors vary depending upon the percentage you are looking at, so to simplify things, pollsters report the maximum sampling error, which occurs when you see a percentage of 50%.  

When you see a sampling error of +/- 5 percentage points, this means that if 50% of the voters said they support the President's policies, this could be as few as 45% or as much as 55%.  If it really varied by +/- 5%, than the poll result would be interpreted as, as few as 47.5% (50% - 5% or 2.5 percentage points) or as much as 52.5% (50% + 5% or 2.5 percentage points).  This is really different than a range from 45% to 55%, isn't it?

Also, these sampling errors are based on sample size.  For a poll of 1,000 people, the maximum sampling error is +/- 3.1 percentage points.  A poll of 300 people has a LARGER error, +/- 5.7 percentage points.

Another critical thing to understand is that all of these errors are based upon a CHOSEN level of confidence.  In corporate America, most CEO's and CFO's want a higher confidence level that their research results are correct.  So, I typically use a 95% confidence level when reporting results to them.  This means that, if I repeat my survey 100 times, in 95 out of 100 of those surveys, the results I give them will vary no more than the sampling error I report.  My examples above for the surveys of 1,000 and 300 people are sampling errors calculated at the 95% confidence level. 

Crafty researchers often use LOWER confidence levels to reduce the amount of sampling error they say is associated with their surveys.  For example, using the example of 1,000 voters, at the 95% confidence level, my maximum sampling error was +/- 3.1 percentage points.  At the 90% confidence level, my maximum sampling error falls to +/- 2.6 percentage points.  Same survey results, same number of interviews, but using smoke and mirrors, I've just made my survey results more reliable.

OK, this is starting to look and feel like a textbook, so three big points to remember as you examine that expensive marketing research survey:

1) What is the sampling method?  And, given who is being interviewed, is it satisfactory?

2) What is the question being answered, and do you believe that it is as fair a question as could be crafted?

3) Margins of error are points, not percentages, and you should look for a higher, not lower confidence level (90% or higher) to feel comfortable with the answers you are seeing.