Where Polls Can Mess Up (and What Pollsters Do About It)

With less than a month until Election Day, you may be paying more attention to political polls, and starting to wonder how much you ought to trust them.

Polls are a useful tool for developing a sense of what issues are top of mind for voters and where the race between Vice President Kamala Harris and former President Donald J. Trump stands. But polling isn’t an exact science; it’s susceptible to common errors and baked-in biases. This is particularly true of election polls.

“Polling is often as much, or more, of an art than a science,” said Ashley Koning, director of the Eagleton Center for Public Interest Polling at Rutgers.

In a previous article about understanding polls, I went over what to look for in an individual poll. Here, let’s dive into common errors that challenge pollsters, how they mitigate those errors and how to think about sources of error as you interpret pre-election polls.

Polls’ potential pitfalls

Theoretically, the only way to create a perfect survey of what every American thinks is to poll every American (though even the U.S. census has flaws). Clearly, that is not feasible, so instead, pollsters who want to measure public opinion survey a sample of the U.S. population. “I like to equate it to tasting a pot of soup,” Dr. Koning said. “You know from that one taste of soup how the rest of the pot tastes because it’s getting you a representative flavor of that pot.”

The spoonful-of-soup analogy helps demonstrate how pollsters can design a survey of just 1,000 people with the aim of representing 337 million. But it also shows why sampling is never truly precise.

The limits of contacting only a sample population bring a certain amount of error into a poll. This is called the sampling error, because a sample by definition doesn’t include everyone. The potential degree of this error in a particular poll is reflected in the margin of sampling error, a measurement of how closely the poll reflects the full population.

Consider a poll showing that 44 percent of likely voters had a favorable view of Taylor Swift, with a margin of error of plus or minus 3.8 percentage points, as a New York Times/Philadelphia Inquirer/Siena College poll did last month. That margin means that if you were to conduct this same survey with many samples of the same size, you could expect between 40.2 percent and 47.8 percent of likely voters to have a positive view of Ms. Swift nearly all of the time.

The margin of sampling error is typically calculated with a statistical formula. But the true margin of error for a poll can be twice as large as reported, research has shown. And there are other errors that can afflict polls.

There’s also coverage error, which happens when a sample excludes certain segments of the population it is trying to poll. For example, online-only polls exclude Americans who don’t have access to the internet. There’s measurement error, which can occur when questions are worded in a way or presented in an order that affects the responses — a good reason to take a peek at exact question wording when you evaluate a poll. And there’s nonresponse bias, in which a certain segment of the population may be less likely to participate in a poll — something pollsters suspect may have happened with supporters of Mr. Trump in 2016 and 2020.

“Ultimately, the worst error a pollster can make is to not have a good model of the population in their sample,” said Scott Keeter, a senior survey adviser at Pew Research Center. If pollsters do a poor job collecting their sample, they may end up over- or underestimating support for a candidate.

How good pollsters adjust

One way pollsters work to reduce sampling error is by adjusting their findings to better match the population they’re trying to measure. This process is called weighting. If a pollster surveyed a population that was 50 percent women and 50 percent men, for example, but their sample included only 40 percent women, they would “weight” the women’s responses more highly to match the target population more closely.

Pollsters use complex statistical equations to weight for many different metrics, such as age, sex and race. Some are now weighting responses based on more variables than they used in 2016 and 2020. For example, in 2016, many pollsters did not weight based on respondents’ education levels, which contributed to an underestimation of support for Mr. Trump. Many pollsters now consider education one of the most important variables for which to weight their data.

High-quality pollsters, such as the “select” ones highlighted on The New York Times’s page of poll averages, take great care to try to mitigate potential errors. They do this through careful survey design and by trying to reach a large and representative sample. Ultimately, though, pre-election polls will always be at least slightly off, because the population they’re trying to measure — the people who will vote — can’t be fully known until Election Day.

The bottom line: Don’t expect perfection

None of this is to say that you should shun polls outright. Despite notable misses in 2016 and 2020, election polls overall have grown more accurate over time, and the ones from 2022 were some of the most accurate. But it’s helpful to consider the minefield of errors that pollsters have to dodge when trying to capture how voters feel before an election.

Keep the margin of error in mind, and remember that the margin doubles when you consider the difference between two findings, such as between two candidates in a race.

If you examine a poll’s breakdown into subcategories, such as responses by race or age, you also have to consider the margin of error for each one. These breakdowns are called cross-tabulations (or cross-tabs), and because they show subgroups with smaller sample sizes, the margins of error are greater. This, combined with the fact that most cross-tab subgroups aren’t weighted, means that looking at cross-tabs for a single survey is limited in what it can tell you.

As high-quality polling has become more expensive and time-consuming, and as presidential races have become tighter, these potential errors have come to the forefront. Keep all this in mind as you study the polls in the final weeks of election season, and you may be less shocked by the results after Election Day.