The problem with polls

Political prognostication has become increasingly unreliable. Why? Here's everything you need to know:

Are polls really less accurate?

There's no doubt about it. In recent years, polls have been egregiously wrong in several high-profile elections. In the months before the 2012 presidential election, an average of leading polls showed a virtual tie between Obama and Mitt Romney, and some — notably Gallup — predicted a narrow Romney victory (see below). Instead, the president easily won re-election, by 5 million votes and a 51.1 percent to 47.2 percent margin. Pollsters vastly underestimated a Republican wave in the 2014 midterms, and last month had an epic fail in the Michigan Democratic primary, with the poll average predicting Hillary Clinton would crush Bernie Sanders by 21 points. When Sanders upset Clinton by 1.5 percent, polling aggregator Nate Silver of FiveThirtyEight.com — who had given the Vermont senator just a 1 percent chance of winning — deemed it "among the greatest polling errors in primary history." And it's not just an American problem. Pollsters totally misread the 2014 referendum on Scottish independence, which was resoundingly rejected, and missed decisive victories last year for Britain's Conservatives and Israeli Prime Minister Benjamin Netanyahu's Likud Party.

The Week

Escape your echo chamber. Get the facts behind the news, plus analysis from multiple perspectives.

SUBSCRIBE & SAVE

Sign up for The Week's Free Newsletters

From our morning news briefing to a weekly Good News Newsletter, get the best of The Week delivered directly to your inbox.

From our morning news briefing to a weekly Good News Newsletter, get the best of The Week delivered directly to your inbox.

Sign up

What's gone wrong?

Pollsters primarily blame recent failures on two factors: "the growth of cellphones and the decline in people willing to answer surveys," says political scientist Cliff Zukin, former president of the American Association for Public Opinion Research. Ten years ago, about 6 percent of Americans relied primarily on cellphones; by 2014 that figure had jumped to 60 percent. That caused problems for opinion researchers, who typically polled by making automated "robocalls" to random landline exchanges and then, when people picked up, passing them to a live interviewer. "To complete a 1,000-person survey, it's not unusual to have to dial more than 20,000 random numbers," Zukin says. Federal law, however, prohibits autodialing cellphones — which means paid interviewers have to make calls manually, which can be prohibitively time-consuming and expensive. As a result, some organizations make compromises, such as leaning too heavily on landline surveys, which can skew results.

How so?

The poll can end up ignoring large segments of the population. "Guess who answers the [landline] phone now? It's all people over 50," says Republican pollster Bill McInturff, who helps conduct the NBC News/Wall Street Journal survey. That can make for a more affluent and conservative sampling. In addition to overlooking younger voters, landline polling also gives short shrift to minorities and the poor, sectors most likely to rely on cellphones. Underrepresenting those groups, McInturff explains, "means you are systematically undercounting Democrats." Since people take their cellphone numbers with them when they move, it's also harder to use area codes to target specific regions. One of the ways pollsters try to compensate for these problems is by "weighting" their results.

What's weighting?

If a polling sample includes 3 percent African-Americans, but African-Americans account for 12 percent of the population, the pollster will "weight" the preferences of the black respondents four times as much. "The goal is noble," says pollster Jeanne Zaino, but weighting "is fraught with challenges and uncertainties. How do we know if the African-Americans sampled represent the views and attitudes of all African-Americans?" Besides race, polls also weight by party, cellphone use, gender, and other factors — but the formula for doing so varies from pollster to pollster, and is subject to error and partisan bias. Rasmussen Reports, for example, is known to skew Republican, while Public Policy Polling (PPP) leans Democratic. Weighting can lead to another phenomenon that affects polling firms' accuracy: "herding." If most polls show a candidate with a 10-point lead, and Poll X finds that the race is much closer, Poll X often finds an excuse to cook its numbers to avoid being an outlier. And no matter how polls are weighted, they really can't compensate for declining response rates.

How far have those rates fallen?

A lot. "In the late 1970s, we considered an 80 percent response rate acceptable," says Zukin. "By 2014, the response rate had fallen to 8 percent." For more than a century, people answered their landline phones faithfully, but they've grown much warier. "Telemarketing poisoned the well," says Charles Franklin, director of the Marquette University survey. Scott Keeter, who runs the respected Pew Research poll, says people can now use voicemail and caller ID to ignore calls from unknown numbers. Online polls are becoming more prevalent, but since they're voluntary, people who choose to answer them may be unusually ideological, skewing the results.

Is there hope for polling?

The most accurate technique appears to be the kind of poll averaging conducted by RealClearPolitics.com and Silver's FiveThirtyEight.com. Silver weights polls according to historical accuracy and factors in demographics and other data to create election models. Despite his miss in Michigan this year, he's scored spectacular successes, correctly calling the outcomes of 49 states during the 2008 presidential election, and a perfect 50 in 2012. Still, as Sean Trende, senior elections analyst for RealClearPolitics​.com, points out, even these data-savvy methods can fail if they're based on flawed samplings of voters. "Electoral modelers have a nerdy little secret," he says. "We aren't oracles."

Gallup's missed call in 2012

Mitt Romney was so sure he would be elected the nation's 45th president in 2012 that he ordered a fireworks display to be unleashed over Boston Harbor the moment he notched his 270th electoral vote. Internal surveys gave him a consistent lead over President Obama, and so did several outside pollsters, including venerable Gallup. But skies over Boston remained dark that Election Night, as Obama cruised to a second term. What went wrong? Gallup's post-mortem found it had misidentified likely voters, undercounted Democratic-leaning regions, overcounted whites, and when calling landlines dialed only listed numbers, which skewed older and Republican. Gallup has tweaked its model for 2016. "When the next presidential election rolls around," promises Gallup's Frank Newport, "we think we'll certainly be in a position to be at the accurate end of the spectrum."