With Election Day fast-approaching, tension in the United States is quickly rising to an absolute climax. Indeed, after a year-long buildup to what could be the most consequential presidential elections in a very long time to come, November 8th is just around the corner. Needless to say, these elections concern so much more than just the United States alone, and it’s for this reason that we’ve decided to write a short series of articles on them, starting with one on the predictive power of polls. Do they ever get it wrong at all? As Election Day draws ever nearer, this question is becoming more and more pressing, so let’s find out what the data suggests.
A brief history of polling
Believe it or not, but political forecasting goes back as far as 1824, when the first known example of an opinion poll — a local straw poll by a newspaper from Pennsylvania — was conducted showing Andrew Jackson leading John Quincy Adams by 335 to 169 in the contest for the United States Presidency. Since Jackson won the popular vote in that state (and indeed the whole country), such straw votes gradually became more popular, but they remained local, usually city-wide phenomena. In 1916, The Literary Digest (an influential weekly magazine) embarked on a national survey and correctly predicted Woodrow Wilson’s election as president. Mailing out millions of postcards and simply counting the returns, The Literary Digest correctly predicted the victories of Warren Harding in 1920, Calvin Coolidge in 1924, Herbert Hoover in 1928, and Franklin Roosevelt in 1932. Then, in 1936, its 2,3 million voters constituted a huge sample, but they were generally more affluent Americans who tended to have Republican sympathies. The Literary Digest was ignorant of this new bias; the week before election day, it reported that Alf Landon was far more popular than Roosevelt. At the same time, George Gallup conducted a far smaller (but more scientifically based) survey in which he polled a demographically representative sample. Gallup correctly predicted Roosevelt’s victory. The Literary Digest soon went out of business, while polling took off.
Nearly two centuries later, polls are still as relevant as ever, but how accurately do they still reflect the public opinion on political matters in today’s tumultuous world? Over the past two years, election polling has seen some spectacular misjudgments. Several organisations tracking the 2014 midterm elections did not catch the Republican wave that led to strong majorities in both houses; polls in Israel badly underestimated prime minister Netanyahu’s strength in 2015; and just recently, pollsters in Great Britain predicted a comfortable win for the Remain camp in the already historic Brexit referendum. All three of these examples reminded the world that pollsters can be awfully mistaken, so as much as we’d all like to (come on, let’s not kid ourselves here), we shouldn’t take Hillary Clinton’s comfortable lead over Donald Trump in the most recent polls for granted just yet.
A few important trends
What’s causing the seemingly increasing unreliability of polls? It appears that two main trends are driving this process: the growth of cellphones, and the decline in people willing to answer surveys. Coupled, they’ve made high-quality research much more expensive to do, and so there’s less of it. This has opened the door for less scientifically based, less well-tested techniques. To top things off, a perennial election polling problem — how to identify likely voters — has become even thornier.
In terms of speed, the growth of cellphones is like few innovations in history. About 10 years ago, opinion researchers began taking seriously the threat that the advent of cellphones posed to the established practice of polling people by calling landline phone numbers. At that time, an estimated 6% of the public only ever used cellphones. For the first half of 2014, this had grown to 43%, with another 17% mostly ever using cellphones. In other words: a landline-only sample conducted for the 2014 elections would miss about three-fifths of the American public — almost three times the amount that it would’ve missed in 2008. Hence, in order to complete a 1,000-person survey, it’s not unusual for pollsters to have to dial more than 20,000 random numbers, some of which don’t even exist anymore. Dialing manually for cellphones takes a great deal of paid interviewer time, and pollsters also compensate cellphone respondents with as much as 10 dollars for their lost minutes. The best survey organisations complete about two of the more expensive cellphone interviews for every one on a landline, and for many organisations this is a budget buster that leads to compromises in sampling and interviewing. Inevitably, the quality of the research then drops.
The second unsettling trend is the rapidly declining response rate. Back in the late 1970s, an 80% response rate was acceptable, and even then researchers would still worry about the 20% that they had missed. By 1997, because of answering machines and other technologies, the response rate had dropped to 36%, and the decline has accelerated; by 2014, the response rate had fallen to an all-time low of just 8%. This decline is worrisome for two reasons. First, of course, is representativeness. For some reason, well-done probability samples seem to have retained their representative character despite the meager response rate, but the risk of surveys failing to reflect the facts obviously increases as the rates drop. The risk isn’t always realised, but with very low response rates now common, we should expect more failed predictions based on these surveys. Second, low response rates have also had a significant impact on the cost. Survey organisations have to pay interviewers to complete between 700 and 1,000 cellphone interviews with a response rate of 8%, with multiple callbacks to numbers that don’t answer (or ineligibly young people). This means tens of thousands of calls dialed by hand, where not that long ago automatic dialers achieved a 100% landline sample.
Of course, news budgets for the media organisations that underwrite so much of this research have also shrunk. This new reality has driven many election pollsters to the internet, where expenses are just a fraction of what it costs to do a good telephone sample. However, there are obviously major problems with internet polls. The first is what pollsters call ‘coverage error’: the simple fact that not everyone can be reached online, as estimates show that ‘only’ 87% of American adults are internet users. However, since internet use correlates inversely with age and voting habits, it makes this a more severe problem in predicting elections. While all but 3% of those aged 18 to 29 use the internet, they made up just 13% of the 2014 electorate. Some 40% of those aged 65 and older doesn’t use the internet at all, yet they made up 22% of all voters.
The last big problem with election polling, albeit not a new one, is the fact that survey respondents often overstate their likelihood of voting. It’s not uncommon for 60% of all potential voters to report that they definitely plan to vote in an election in which only 40% of those people actually do. Pollsters thus have to guess who’ll actually vote, and organisations construct ‘likely voter’ scales from respondents’ answers to a bunch of questions about the election. However, research shows that there’s no single magic-bullet question (or set of questions) that correctly predicts voter turnout yet, and even if there was, people might still lie about their choice, possibly out of shame.
Will Donald trump Hillary?
The conclusion, then, is that polls haven’t become much more reliable at all since the last big presidential rundown in 2008; in fact, they might well have become less reliable, since it has gotten more and more difficult to draw a perfectly random sample of the population. Still, we humans are curious beings, and a sneak peek surely can’t hurt, right? A quick glance at BBC’s US election poll tracker, which looks at the five most recent national polls and takes the median value, tells us that Clinton has a comfortable lead over Trump — a solid 5% at the time of writing. Since Trump must win in most swing states in order to stand a serious chance of winning, the White House still seems far away for him. However, if there’s anything that this man has shown, it’s that you’ve only defeated him when the game’s really over. Bring it on — it’s gonna be a cracker.
Gallup’s record in presidential elections is 85%, correctly predicting 17 out of the last 20 winners.