What to Look Out For
At TVPoll, we think it is important to provide the public with information about polls they may encounter that are not based on solid, scientific methods. An informed public should have the information it needs to distinguish between questionable polls and polls they can rely on.
We often receive calls and emails about the results of various polls reported by the media. People want to know if they can take the results of these polls seriously. Were they really based upon solid scientific research methods? With the help of several research associations, we have compiled information about some non-scientific polls of which an informed public should be wary.
Science or Entertainment?
Many people these days are using the term “poll” or “survey,” but these terms may not always be used these days to indicate a scientifically-conducted poll or survey.
Do not be fooled.
One good indication is if a newspaper article doesn’t report the sample size and the margin of error of the poll, then it was NOT conducted scientifically and was printed for “entertainment” purposes only. Many times, the sample sizes are very small and any conclusions should be highly questionable.
This is an injustice to the typical reader who might believe the poll was a scientific poll, but nonetheless some disreputable pollsters will try and make it look that way.
Internet Surveys
A growing trend in public opinion polls has been that of Internet surveys. For the most part, these surveys are not conducted in a scientific manner. Results of such polls are often misleading.
Because there are many Americans who do not have access to or are not regular users of the Internet, online polls that claim to measure public opinion in a scientific manner can hardly be expected to do so with accuracy.
Another problem with online polling is the fact that respondents decide for themselves whether or not to participate. In a scientifically valid poll, respondents are targeted by a carefully designed sampling process. Also, people may choose to respond more than once to an Internet poll, further compromising its validity.
To explore the challenges in-depth to polling on the Internet, read TVPoll’s Pitfalls in Online Polling.
Also, Read the American Association of Public Opinion Research’s statement about online polling.
Push Polls
A push poll is where, using the guise of opinion polling, disinformation about a candidate or issue is planted in the minds of those being ‘surveyed’. Push polls are designed to shape, rather than measure, public opinion. A type of political telemarketing, push polls are only effective if they can call more voters than just a representative sample (the goal of legitimate pollsters).
But, not all questions that seem negative are part of push polls. As Kathy Frankovic, Director of Surveys for CBS News noted, “Candidate organizations sometimes do actual polls that contain negative information about the opposing candidate. These polls, which are not push polls, are conducted for the same reasons market and advertising researchers do their work: to see what kinds of themes and packages move the public”.
In the advertisers’ case, they want to figure out the best way to reach buyers; candidate pollsters need to motivate voters. Polls done for campaign research are full-length, with more topics than just questions about the opponent, and include demographic questions that allow researchers to categorize respondents. Interviewers will not ask to speak to anyone by name, but are calling a sample of randomly selected telephone numbers.
How do you distinguish a push poll from the legitimate poll? According to Mark Blumenthal, the Mystery Pollster, the proof is in the intent: If the sponsor intends to communicate a message to as many voters as possible rather than measure opinions or test messages among a sample of voters, it qualifies as a “push poll”.
We can usually identify a true push poll by a few characteristics that serve as evidence of that intent. “Push pollsters” (and Mystery Pollster hates that term) aim to reach as many voters as possible, so they typically make tens or even hundreds of thousands of calls. Real surveys usually attempt to interview only a few hundred or perhaps a few thousand respondents (though not always).
Push polls typically ask just a question or two, while real surveys are almost always much longer and typically conclude with demographic questions about the respondent (such as age, race, education, income).
The information presented in a true push poll is usually false or highly distorted, but not always. A call made for the purposes of disseminating information under the guise of survey is still a fraud – and thus still a “push poll” – even if the facts of the “questions” are technically true or defensible.
Accusations of push polling have become a political attack in itself in recent elections with candidates attacking each other over polls their opponent’s pollster conducted that asked negative questions. In most cases the polls were legitimate, but each side decided to take advantage of the situation to merely attack their opponent. In the end, the listening public is typically left in doubt about what was fraud and what was legitimate , and more skeptical and less willing to participate the next time legitimate pollsters, like SoonerPoll, come calling.
For the record, TVPoll is a public opinion research firm and does NOT engage in push polling of any type.
University-Organized Surveys for Class Credit.
Surveys conducted by university students as part of a class assignment can be flawed for a number of reasons.
For example, in January 2004, the Modesto Bee reported that a professor at California State University, Stanislaus, required students in his class to conduct public opinion interviews by telephone to receive credit for a course, but failed to provide resources and oversight or to validate that the students carried out the interviews.
The Bee reported that student interviewers received only an hour of training and were expected to pay for long distance charges. Several students came forward to say they falsified their interviews because they were pressed for time and would have to make lengthy long distance telephone calls at their own expense, according to the story. The falsified results of the public opinion poll were introduced as evidence in the Scott Petersen murder trial for possible change of venue in the trial.
Polls You Can Count On
TVPoll is a member of the American Association for Public Opinion Research and abides by the Code of Professional Ethics and Practices as set out by that organization. Since SoonerPoll does not outsource any aspect of the polling process, we are able to ensure that our high standards are met throughout the duration of every research project we conduct. It is through the consistent maintenance of those high standards that SoonerPoll has emerged as one of the most trusted, respected researchers in the industry.
By bshapard, April 9th, 2010,in Uncategorized » | 16,676 Comments »
Before publishing any poll, here is a list of the 9 questions every journalist should ask before reporting the results.
Who conducts the poll?
Was it a reputable, independent pollster? If not, then regard its findings with caution. If you are not sure, then one test is its willingness to answer the questions below. Reputable pollster should provide you with the information you need to evaluate the survey.
Who paid for the poll and why was it done?
If it was conducted for a respected media outlet, or for independent researchers, there is a good chance it was conducted impartially. If it was conducted for a partisan client, such as a company, interest group or political party, it might still be a good survey (although readers/listeners/viewers should be told who the client was). The validity of the poll depends on whether it was conducted by an organization that used a scientific approach to sampling and questionnaire design, whether it asked impartial questions, and whether full information about the questions asked and results obtained are provided.
If such information is provided, then the quality of the survey stands or falls according to its intrinsic merits. If such information in not provided, then the poll should be treated with caution. In either event, watch out for loaded questions and selective findings, designed to bolster the view of the client, rather than report public opinion fully and objectively.
What was the sample size for the survey?
The more people, the better – although a small-sample scientific survey is ALWAYS better than a large-sample self-selecting survey. Note, however, that the total sample size is not always the only relevant number. For example, voting intention surveys often show figures excluding “don’t knows”, respondents considered unlikely to vote, and those who refuse to disclose their preference. While excluding these groups ensures that, the poll reports the opinion of the most relevant group – “likely voters” — reported voting-intention sample size may be significantly lower than the total sample, and the risk of sampling error therefore greater.
How were those respondents chosen?
Is it clear who is included in the sample and who was left out? If the poll claims to represent the public as a whole (or a significant group of the public), has the polling company employed one of the methods outlined in the questions above? If the poll was self-selecting – such as readers of a newspaper or magazine, or television viewers writing, or certain web surveys – then it should NEVER be presented as a representative survey. If the poll was conducted in certain locations but not others, for example, cities but not rural areas, then this information should be made clear in any report.
When was the poll done?
Events have a dramatic impact on poll results. The interpretation of a poll should depend on when it was conducted relative to key events. Even the freshest poll results can be overtaken by events. Poll results that are several months old may be perfectly valid, for example, if they concern underlying cultural attitudes or behaviors rather than topical events, but the date when the poll was conducted (as distinct from published) should always be disclosed. The date of the fieldwork is particularly important for pre-election polls where voting intention can change right up to the moment the voter records their vote.
How were the interviews conducted?
There are four main methods; in person, by telephone, online or by mail. Each method has its strengths and weaknesses. Telephone surveys do not reach those who do not have telephone. Online surveys reach only those people with internet access. All methods depend on the availability and voluntary cooperation of the respondents approached; response rates can very widely. In all cases, reputable companies have developed statistical techniques to address these issues and convert their raw data into representative results.
What were respondents asked?
Try to get a copy of the full questionnaire, not just the published questions. A reputable organization should publish the questionnaire on its website, or provide it on request. Decide if the questions were balanced and be cautious about the results if the interview was structured in a way which seemed to lead the respondent towards a particular conclusion.
Are the results in line with other polls?
If it is possible, check other polls to see if the results are similar or very different. Surveys aiming to cover the same topic should come to similar conclusions. If the answers are very different, the reasons may become apparent when the questionnaire or the sampling method is examined.
Has the pollster met the minimum disclosure requirements and abided by the code of best practices and ethical standards set out by the public opinion polling industry?
Reputible pollsters have pledged to maintain high standards of scientific competence and integrity in conducting, analyzing, and reporting their work; in their relations with survey respondents (such as observing the Respondent’s Bill of Rights); with their clients; with those who eventually use the research for decision-making purposes; and, with the general public. They have also pledged to reject all tasks or assignments that would require activities inconsistent with the principles of this code.
For more information on polling best practices and the code for professional ethics, see the AAPOR website.
By bshapard, April 9th, 2010,in Uncategorized » | 43,665 Comments »