Are surveys useless for predicting what people will do because people can't be trusted to tell the truth?
It's a common and recurring sentiment. Jason Oke, then a planner at Leo Burnett, blogged about it way back in 2007. Faris Yakob threw a bomb of a blog post in 2010. Richard Shotton wrote about it in 2017.
The concerns about respondents' biases are as legitimate as they are well known. The problem even has a name: "can't say / won't say." How to ask questions in a way that produces reliable data is quite literally a science. It has its own experiments, discoveries, and textbooks.
Many factors affect the way people respond to surveys. The Asking Questions book outlines a few of them:
- comprehension of survey questions
- the recall of relevant facts and beliefs
- estimation and inferential processes people use to answer survey questions
- the sources of the apparent instability of public opinion
- the difficulties in getting responses into the required format.
Philip Graves, "a consumer behaviour consultant, author and speaker," takes a dim view of market research surveys in his book Consumerology. Among other things, Graves writes that "attempts to use market research as a forecasting tool are notoriously unreliable, and yet the practice continues."
He then uses political polling as an example of an unreliable forecasting tool. He does not elaborate beyond this one paragraph:
Opinion polls give politicians and the media plenty of ammunition for debate, but nothing they would attach any importance to if they considered their hopeless inaccuracy when compared with the real data of election results.
It's worth taking a closer look at political polls for two reasons.
First, horse race polls ask exactly the forward-looking "what will you do" kind of question that people, presumably, should not be able to answer in any reliable way. Here's how these questions usually look; this example is from Pew's 2012 questionnaire (pdf, methodology):
If the presidential election were being held TODAY, would you vote for
- the Republican ticket of Mitt Romney and Paul Ryan
- the Democratic ticket of Barack Obama and Joe Biden
- the Libertarian Party ticket headed by Gary Johnson
- the Green Party ticket headed by Jill Stein
- other candidate
- don’t know
Second, in election polling, there's nowhere to hide. The data and the forecasts are out there, and so, eventually, are the actual results.
And so, every two and four years, we all get a chance to gauge how good surveys are at forecasting people's future decisions.
How Accurate Are Political Polls?