Interpreting Polls and Opinion Research: A Researcher's Briefing

Political pundits, politicians and pollsters use opinion polls and other forms of marketing and social research to investigate opinions, make policy and design strategy.  Oh, and they use them as convenient information when the data support their points of view.

As a marketing researcher for 20+ years, I spend a great deal of time helping people design studies, field and interpret them to better understand their business and other objectives.  Despite the fact that political and opinion polls are generally the fodder of news organizations, think tanks, NGO's and political groups, the principles underlying consumer research remain the same.

Figures lie and liars figure, right?

Not always.  Being skeptical of the methods of political and opinion polls is healthy.  Those conducting these investigations often use inappropriate methodologies, poor sampling, bad questionnaire design and atrocious analyses.  But being a critical consumer of opinion polls will help you not only be smarter in debates about the results, but also a better voter as well.

Let's start with sampling methods. 

Polls were originally conducted face-to-face, door-to-door at people's homes.  Social scientists used Census data to figure out how many people were needed from each Census tract to make a representative sample.  But, with the rise of crime in many areas, pollsters walking door to door were presented with many, um, challenges, in recent years.

Widespread telephone penetration in the middle of the last century led to the increased use of telephone polls.  Most people had a telephone, and using AT&T's complex exchange assignment algorithms, pollsters were then able to calculate who lived in which areas, and essentially replicate what Census block walkers used to do with door to door methods.  Even de-listing your phone number won't avoid having you being called by sophisticated polling firms, since most were able to randomly generate telephone numbers using the exchange information they received from Ma Bell and her little Bellettes.

Needless to say, abusive direct marketing practices have made telephone surveying challenging (despite the fact that politicians have cleverly left polling out of most legislation designed to protect people from unwanted interruptions at home), but this is still the most effective way to reach consumers in most countries today.

The increasing penetration of home computers and internet usage has created opportunities to poll people in panels of voters, designed by social scientists to be nationally representative.  Online polling has increased in popularity as well, but today, is still somewhat less representative than telephone polling.  Consider that many groups are still very underserved with online access (African Americans, unaculturated Hispanics, etc.), and others are slow to adopt online practices (e.g., seniors, less well educated, etc.).

These differences are decreasing, however, as online usage becomes more pervasive nationally.

Other surveying methods like mail, and combination methods like postcards directing voters to an online survey or an automated telephone survey suffer from low response rates (making them expensive), and are also associated with a source of polling error called "non-response bias."  All survey methods suffer from this, and the only solution a pollster has is to have good information upfront about what a representative sample would look like, and endeavor to fill quotas to ensure that the poll remains representative.

When you first look at an opinion poll, look for the sampling method.  Was it a telephone survey, online poll or mail method?  If it was telephone, know that with over 100 million telephone equipped homes in the US, a reputable pollster had lots of possible people to sample from.  If it was an online or internet poll, be slightly more suspicious.  There are no national directories of email addresses yet, so there is no way to sample randomly from among all online users.  AOL may know who all of their members are, but that leaves out all non-AOL users.  Online panels are often recruited by other means (calling or email-blasting people), and reputable pollsters will often have access to panels that are designed to be nationally representative. 

However, it is still likely that certain groups will be under-represented in these polls, like the seniors or non-English speaking Hispanic population.  Even stacking the panel full of sought-out seniors or Hispanics poses a methodological problem:  if most people in these groups are not online, those who are, are innately different than everyone else.  They are more sophisticated technologically, usually better educated or have higher incomes.  Ask yourself whether you believe that these people really represent the people they are supposed to.

Survey design, including writing a decent questionnaire, is the one area of polling that is usually most suspect.  When reviewing a poll, if the question isn't listed, wonder why.  How are you supposed to interpret the answers if the question isn't known?  Consider the veracity of the questionnaire writer.  Is he or she likely a partisan political pundit?  If so, you can bet that the questionnaire is going to be written to address a particular point of view rather than be investigatory like many marketing research surveys are.

A recent poll I participated in had the following question: "How much would you say you support the President's policies in the Middle East?  Would you say that you support them fully, support them somewhat, or are you a whining liberal Communist, trying to wreak destruction and death on all true Americans?"  I'm exaggerating, of course, but the poll was written to obtain the pollster's point of view, not learn how I feel about the situation.

It's also important to look at the types of answers given to a question.  In a rating question, like my silly example above, one of the answers is often "Don't Know/Not sure." While that may be appropriate, whether or not the pollster leaves in the people who answer this is sometimes not clear.  Leaving in the Don't Know answers is completist, but it sometimes deflates the attitude ratings you're looking for.  Many pollsters will report the percentage or number of people who don't have an opinion, and then report how those who DID have an opinion answered.  That's a better way to handle that.

Finally, there is the often reported margin of error associated with polls.  A "margin of error" is a statistical way of estimating the stability of the results.  This number comes from a formula based upon the number of people who participated in the poll, sometimes the size of the population (more important for small universes, like Communists living in Wisconsin, or PETA members in Manhattan), the actual percentages found in a survey question.

Many polls simply report the MAXIMUM error associated with the sample size of the poll.  This is often reported as +/- 5%.  First, know that sampling error (the real name of this statistic, by the way) is actually reported in PERCENTAGE POINTS, not percent.  So, the right way to give the sampling error in my example is +/- 5 percentage points.  Sampling errors vary depending upon the percentage you are looking at, so to simplify things, pollsters report the maximum sampling error, which occurs when you see a percentage of 50%.  

When you see a sampling error of +/- 5 percentage points, this means that if 50% of the voters said they support the President's policies, this could be as few as 45% or as much as 55%.  If it really varied by +/- 5%, than the poll result would be interpreted as, as few as 47.5% (50% - 5% or 2.5 percentage points) or as much as 52.5% (50% + 5% or 2.5 percentage points).  This is really different than a range from 45% to 55%, isn't it?

Also, these sampling errors are based on sample size.  For a poll of 1,000 people, the maximum sampling error is +/- 3.1 percentage points.  A poll of 300 people has a LARGER error, +/- 5.7 percentage points.

Another critical thing to understand is that all of these errors are based upon a CHOSEN level of confidence.  In corporate America, most CEO's and CFO's want a higher confidence level that their research results are correct.  So, I typically use a 95% confidence level when reporting results to them.  This means that, if I repeat my survey 100 times, in 95 out of 100 of those surveys, the results I give them will vary no more than the sampling error I report.  My examples above for the polls of 1,000 and 300 people are sampling errors calculated at the 95% confidence level. 

Crafty pollsters often use LOWER confidence levels to reduce the amount of sampling error they say is associated with their polls.  For example, using the example of 1,000 voters, at the 95% confidence level, my maximum sampling error was +/- 3.1 percentage points.  At the 90% confidence level, my maximum sampling error falls to +/- 2.6 percentage points.  Same survey results, same number of interviews, but using smoke and mirrors, I've just made my poll results more reliable.

OK, this is starting to look and feel like a textbook, so three big points to remember as you examine that poll:

1) What is the sampling method?  And, given who is being polled, is it satisfactory?

2) What is the question being answered, and do you believe that it is as fair a question as could be crafted?

3) Margins of error are points, not percentages, and you should look for a higher, not lower confidence level (90% or higher) to feel comfortable with the answers you are seeing.