Interview Effects

The final source of potential innacuracy in public opinion polling that we'll consider in this course is the administration of a poll -- that is, the manner in which a questionnaire is presented to respondents and their answers are recorded.

Self-Administered Polls

Self-administered entail respondents reading the questionnaire on their own and recording their own answers. Online polls are a good example of self-administered polls, as are polls completed via snail mail and the teaching evaluations you fill out at the end of each course at ISU.

The major "pro" of self-administered polling should be fairly obvious in light of what you've already learned about respondent insincerity: A pollster can generally expect more honesty from respondents when they don't have to share their answers out loud with another person whom they fear might make judgements about thier beliefs and behaviors. There are some downsides as well, though. For example, self-administered polls (especially mail-in polls) tend to have a lower response rate than polls that are administered by an interviewer (see below). In addition, open-ended questions, or questions that ask respondents to articulate answers in their own words rather than choose them off a multiple choice list, are less likely to be answered -- let alone answered at length -- in self-administered polls. This limits the amount of information that can be gathered from public opinion polls and also makes it harder to discern when respondents are offering nonattitudes rather than actually-held opinions.

Interviewer-Administered Polls

In contrast to self-administered polls, interviewer-administered polls make use of a "middle-man" who reads the questionnaire to respondents and records answers on their behalf. Telephone polls (which are a very efficient way to conduct a large-scale poll, and so are very commonly used in the polling industry) are are the most typical form of interviewer-administered polls, but there are also in-person polls (which was the "state of the art" back in polling's early days) in which the interviewer and respondent interact face-to-face.

The pro's and con's of interviewer-administered polling are basically the flip-side of self-administered polling. On the one hand, it heightens the likelihood of insincerity; on the other, it improves response rates and enables pollsters to gather more extensive information through open-ended questions. Well-trained interviewers can also help clarify when respondents are confused about a question's wording or response alternatives.

Trait of Interviewer Effects

A lot of public opinion research focuses on a polling administration problem that emerges only in interviewer-administered polls: trait of interviewer effects, or biased results due to the interviewer's personal characteristics, including especially their race and gender. Trait of interviewer effects are the topic of the last required text for this unit, a video interview with CBS Deputy Director of Surveys Sarah Dutton at last summer's AAPOR conference about race and gender of interviewer effects in the 2008 primaries:

...Read more

Questionnaire Effects

So far, we've seen two potential sources of invalidity and unreliability in public opinion polls: (1) using sampling techniques that result in an unrepresentative samples, and (2) respondents misrepresenting their opinions, either by offering opinions when they don't in fact have any (i.e. nonattitudes) or by giving responses that they believe to be most socially desirable whether or not they're their true opinions (i.e. insincerity). This post considers a third possibility: The design of the questionnaire itself, which is addressed in the second required text for this unit, AAPOR's "Question Wording" FAQ page.

There are basically aspects of a questionnaire that can trip pollsters up: its question order, its question wording, and its response alternatives.

Question Order

The AAPOR reading covers this pretty well; to recap in a nutshell, problems can emerge when questions that appear early on in a questionnaire affect the way in which respondents answer questions that appear later on. A good example comes from some studies conducted during the 1980s, in which researchers looked at responses to the following two questions:

• Do you think a Communist country like Russia should let American newspaper reporters come in and send back to America the news as they see it?


• Do you think the United States should let Communist newspaper reporters from other countries come in here and send back to their papers the news as they see it?

They found that when the questions were asked in this order, significantly more people said they'd support communist reporters coming to the US than when the questions were asked in the reverse order. This is likely because respondents who had already expressed support for Russia to allow access to American reporters would have felt hypocritical if they wouldn't support a similar policy in the United States.

Question Wording

There are many ways for question wording to go wrong, as the AAPOR reading suggests. Leading questions, which (as the name suggests) lead respondents towards a particular answer, are a frequent and obvious cause for concern. At least as frequently problematic, though, are questions that are simply imprecise or confusing. For example, a 1992 Roper poll asked respondents the following question:

• Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened?

The poll yielded astonishingly high numbers of people claiming to either not know (12%) or that it seemed possible that the Holocaust never occured (22%). Shortly thereafter, the Roper pollsters re-ran the poll with the question reworded as follows:

• Does it seem possible to you that the Nazi extermination of the Jews never happened, or do you feel certain that it happened?

The result this time around was more along the lines that were initially expected (and corroborated by a similar poll by the Gallup Organization), as just 1% of respondents said it seemed possible that the Holocaust never happened.

Response Alternatives

Finally, as the cartoon on the right suggests, a questionnaire can produce skewed poll results because the "multiple choice" options offered to respondents as possible answers to the poll's questions are inappropriate.

Sometimes, as in the cartoon, the problem is glaringly obvious. More often, though, it takes more careful reflection to discern where bias may ensue -- consider, for example, the response alternatives in some of the questions presented in "War or War?" (pdf), an article that appeared in Public Perspective about a year after the 9/11 terrorist attacks.

There are also yet more subtle ways for response alternatives to be problematic. Take, for example, a question that asks respondents how often they engage in some behavior -- say, working out at a gym -- and offers the following response alternatives: (a) often, (b) sometimes, (c) rarely, (d) never. The problem here is that "often" and "sometimes" mean different things to different people. So, you might have two respondents who both work out twice a week, but one of whom claims to work out "often" and the other claims to work out just "sometimes." And finally, as we saw in the case of non-attitudes, sometimes the absense of a "don't know" or "no opinion" option can lead respondents to take a side that they don't actually believe in.

...Read more

Nonattitudes and Insincerity

While, as we saw in Topic I, public opinion data may be skewed because of inappropriate sampling methodologies, sometimes polls produce inaccurate results because the people who participate them -- even if they comprise a maximally representative sample of the whole population -- aren't quite truthful about their opinions. There are two primary facets to the "respondents lie" problem: nonattitudes and insincerity.

Nonattitudes (sometimes also known as "pseudo-opinions")

Nonattitudes is a term coined by public opinion researchers back in the early days of public opinion polling to refer to the phenomenon of people offering an opinion even when they don't actually have one. Various studies have established that nonattitudes are a real phenomenon. One of the more prominent examples is a study conducted by University of Cincinnati researchers duing the late 1970s, in which they asked respondents their opinions on the "Public Affairs Act of 1975." More than 30% of the respondents offered an opinion, even though there actually is no such legislation.

As you might imagine, when it comes to the prevalence of nonattitudes, not all polling subjects are created equal. Most people do have opinions about high-profile subjects (e.g., abortion, the war in Iraq, Barack Obama). However, when it comes to relatively obscure policies, personalities, and events, nonattitudes are increasingly likely to surface. Similarly, while they may have opinions about an issue in general, they may not have opinions about its finer points and technicalities -- but might nevertheless offer "pseudo-opinions" about them when responding to a public opinion poll. As for why respondents report nonattitudes rather than just decline to respond at all -- there are various possibilities, ranging from wanting to appear more knowledgable than they actually are to feeling pressured to "just answer the question" that's asked of them.

There are a few steps pollsters can take to minimize the problem of nonattitudes creeping into their poll results:

• Screening questions at the start of the poll can weed out likely nonattitudes by asking respondents from the get-go whether they are familiar with the poll's subject and/or have an opinion about it.
• Follow-up questions can also minimize the nonattitudes problem by asking respondents to elaborate on their simple "yes/no," "agree/disagree," or "favor/oppose" responses with explanations as to why they hold those opinions.
• A "mushiness index" (i.e. an index of how "mushy" -- that is, unfixed -- an individual's opinions on some topic are) can be integrated into the poll. This would consist of a series of questions (usually in the neighborhood of 3-5) that ask how much respondents know about the topic, how often they think about or discuss it, and how passionate they are about it.
• Finally, simply providing respondents with the explicit possibility of resopnding "no opinion" or "I don't know" can limit the number of people who offer opinionated responses that they don't actually believe in.

Insincerity

Sometimes respondents do have an opinion about a poll's subject, but choose not to divulge it, as described in this unit's first required text, "When Voters Lie," an article that was published in the Wall Street Journal last summer.

As with nonattitudes, insincerity affects some polling subjects more than others. In contrast to nonattitudes, though, the problem surfaces less as a result of respondents' lack of knowledge about the poll's subject than their psychological urge to come off as "socially desirable" to the poll administrator. So, insincerity is often a problem when people are polled about their behaviors, whether pertaining to their hygiene, their participation in illicit activities, or even whether they vote in presidential elections. People are also more likely to provide insincere responses when asked about attitudes they hold that might be perceived by others as "socially unacceptable" or "politically incorrect." As quite a few of the required texts in this unit suggest, polling subjects that have an explicit racial angle are especially susceptible to bias stemming from voter insincerity.

The simplest way to minimize biased results stemming from insincerity is to have respondents self-administer the poll -- that is, to read the questions and record their responses themselves rather than have an interviewer do it. This isn't always possible, though, especially since telephone polling is often the most effective way to conduct a timely, large-scale poll. Therefore, coming up with new ways to get around the problem of respondent insincerity is always a hot topic in public opinion research. You can hear about one example of an alternative approach in another of this unit's required tests, a Youtube video clip of an interview with an award-winning graduate student presenter at the American Association for Public Opinion Research (AAPOR) conference last summer:


...Read more

Validity and Reliability

It's become something of a fad lately for instructors to have their students create and upload Youtube videos as class assignments; this one from the University of Texas does a pretty good job explaining the concepts of validity and reliability:



To recap in a nutshell, a valid instrument measures what it is intended to measure while a reliable one yields the same measurement outcome every time it's used. Also, while they sometimes go hand-in-hand, validity and reliability in fact vary independently of each other, as this target shooting analogy suggests:

So, why is all this relevant to public opinion?

As I've noted before, public opinion polls are essentially an instrument or tool we use to measure public opinion. Like all measuring devices, the results they produce may or may not be valid or reliable. Indeed, because so many factors go into designing and executing a public opinion poll -- selecting a sample, writing up a questionnaire, administering it, and analyzing the responses -- there's a lot of room for invalidity and unreliability to creep in. And, while the polling industry has made considerable strides towards improvings polling results over the last half-century or so (see the Zetterberg article from Topic I), they're still far from having perfected their craft.

The remainder of Topic II will be devoted to considering various sources of potential inaccuracy in public opinion polling -- ranging from lying respondents to problematic questionnaire design and poll administration -- and identifying some steps pollsters can take to avoid or overcome these pitfalls. ...Read more