Questionnaire Effects

So far, we've seen two potential sources of invalidity and unreliability in public opinion polls: (1) using sampling techniques that result in an unrepresentative samples, and (2) respondents misrepresenting their opinions, either by offering opinions when they don't in fact have any (i.e. nonattitudes) or by giving responses that they believe to be most socially desirable whether or not they're their true opinions (i.e. insincerity). This post considers a third possibility: The design of the questionnaire itself, which is addressed in the second required text for this unit, AAPOR's "Question Wording" FAQ page.

There are basically aspects of a questionnaire that can trip pollsters up: its question order, its question wording, and its response alternatives.

Question Order

The AAPOR reading covers this pretty well; to recap in a nutshell, problems can emerge when questions that appear early on in a questionnaire affect the way in which respondents answer questions that appear later on. A good example comes from some studies conducted during the 1980s, in which researchers looked at responses to the following two questions:

• Do you think a Communist country like Russia should let American newspaper reporters come in and send back to America the news as they see it?


• Do you think the United States should let Communist newspaper reporters from other countries come in here and send back to their papers the news as they see it?

They found that when the questions were asked in this order, significantly more people said they'd support communist reporters coming to the US than when the questions were asked in the reverse order. This is likely because respondents who had already expressed support for Russia to allow access to American reporters would have felt hypocritical if they wouldn't support a similar policy in the United States.

Question Wording

There are many ways for question wording to go wrong, as the AAPOR reading suggests. Leading questions, which (as the name suggests) lead respondents towards a particular answer, are a frequent and obvious cause for concern. At least as frequently problematic, though, are questions that are simply imprecise or confusing. For example, a 1992 Roper poll asked respondents the following question:

• Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened?

The poll yielded astonishingly high numbers of people claiming to either not know (12%) or that it seemed possible that the Holocaust never occured (22%). Shortly thereafter, the Roper pollsters re-ran the poll with the question reworded as follows:

• Does it seem possible to you that the Nazi extermination of the Jews never happened, or do you feel certain that it happened?

The result this time around was more along the lines that were initially expected (and corroborated by a similar poll by the Gallup Organization), as just 1% of respondents said it seemed possible that the Holocaust never happened.

Response Alternatives

Finally, as the cartoon on the right suggests, a questionnaire can produce skewed poll results because the "multiple choice" options offered to respondents as possible answers to the poll's questions are inappropriate.

Sometimes, as in the cartoon, the problem is glaringly obvious. More often, though, it takes more careful reflection to discern where bias may ensue -- consider, for example, the response alternatives in some of the questions presented in "War or War?" (pdf), an article that appeared in Public Perspective about a year after the 9/11 terrorist attacks.

There are also yet more subtle ways for response alternatives to be problematic. Take, for example, a question that asks respondents how often they engage in some behavior -- say, working out at a gym -- and offers the following response alternatives: (a) often, (b) sometimes, (c) rarely, (d) never. The problem here is that "often" and "sometimes" mean different things to different people. So, you might have two respondents who both work out twice a week, but one of whom claims to work out "often" and the other claims to work out just "sometimes." And finally, as we saw in the case of non-attitudes, sometimes the absense of a "don't know" or "no opinion" option can lead respondents to take a side that they don't actually believe in.