How to read a poll before you share it

Scroll past the headline. Find the link to the actual poll, which is usually a PDF with a name like "Topline_Results_Final.pdf," and go to the bottom of the first page. Four things live down there: how many people were interviewed, what kind of people they were, what dates the interviews happened, and who paid for it. If a story about a poll doesn't link to that document, you're reading somebody's summary of somebody's summary, and you should treat it accordingly.

That's the whole discipline. Everything below is just what those four numbers mean.

Sample size sets the floor, not the ceiling

A national poll of about 1,000 adults carries a margin of error of roughly plus or minus three points. That's not a convention somebody picked. It falls out of the math: sampling error shrinks with the square root of the sample, so 1,000 respondents gets you to about ±3, and 400 gets you to about ±5, and 100 gets you to about ±10.

Notice what that means going the other way. To cut your margin of error in half, you need four times the people. A poll of 4,000 isn't four times better than a poll of 1,000. It's about twice as precise, at four times the cost, which is why almost nobody does it.

So when a pundit says a poll of 800 people can't possibly represent 260 million adults, he's wrong about the math and right about the instinct. The sample size isn't the weak point. The weak point is who ended up in the sample.

The margin of error is bigger than the number they print

Two traps here, and news write-ups fall into both constantly.

Trap one: the gap. If Candidate A is at 48 and Candidate B is at 45 in a poll with a ±3 margin, the story will say A leads by three. But the margin applies to each number separately, and the error on the difference between them is larger, close to double. A three-point lead in a ±3 poll is a coin flip wearing a suit. "Statistical tie" is an overused phrase, but this is what it's supposed to mean.

Trap two: the crosstab. The interesting stuff in any poll is buried on page 14, where they break the sample down by age, race, region, religion, income. Somebody pulls out a line that says young men shifted 12 points, and it goes around the internet by lunchtime.

Go look at the sample size for that cell. If the full poll was 1,000 adults, the young-men subgroup might be 90 people. At n=90, the margin of error is around ±10. A 12-point "shift" in a group that small is noise with a good publicist. Subgroup findings are worth reading as a hint about where to look next. They're not worth treating as a fact.

Adults, registered voters, likely voters

This is the single most consequential line in the methodology and the one most often dropped from coverage.

All adults includes people who aren't registered and won't vote. Fine for measuring public opinion on, say, school curriculum. Nearly useless for predicting an election.

Registered voters is closer, and it's the standard population for polls taken a year out.

Likely voters is where the judgment comes in. There's no registry of likely voters. Every pollster builds a screen: self-reported intention, past vote history, interest in the race, sometimes a scored model. Two honest shops using different screens on the exact same raw interviews can produce results several points apart. That's not fraud. That's a modeling choice, and it's the choice that most often explains why two polls in the field the same week disagree.

When you see two polls fighting, check the population line before you check anything else. Half the time the fight dissolves. One asked adults and one asked likely voters, and they're measuring different countries.

Weighting, and why response rates matter now

Almost nobody answers the phone anymore. Response rates for telephone polls have collapsed over the past two decades, which means the people who do pick up are, by definition, unusual. They're more available, often more politically engaged, sometimes more trusting of institutions.

Pollsters correct for this by weighting. If the sample has too few people without a college degree, the ones they got count for more. Standard practice, and necessary. But weighting only fixes the imbalances you choose to weight on. If you weight to age, sex, race and region but not education, and education is driving the vote, you'll be off. That's roughly the story of the state-level polling misses in 2016, and it's why most major shops now weight on education. Post-election reviews after 2020 found errors pointing in a consistent direction again, which suggests something is still missing from the correction.

Here's the part worth internalizing: the margin of error only measures sampling error. It says nothing about weighting choices, likely-voter screens, question wording, or the people who hung up. Those errors don't come with an error bar, and they're usually bigger.

Question wording and order

Read the actual questions. They're in the same PDF, in the order they were asked.

Wording moves answers, sometimes a lot. "Estate tax" and "death tax" describe the same policy and get different responses. "Should government do more to help X" and "should government spend more to help X" aren't the same question. Neither side of the aisle invented this, and advocacy outfits on every side commission polls with questions engineered to produce a press release.

Order matters too. If you ask five questions about crime and then ask what the most important issue facing the country is, crime does well. That's not a conspiracy, it's how memory works, and good pollsters rotate question order to blunt it.

A simple test: could you write the headline before you saw the data? If the question is loaded enough that the answer was never in doubt, the poll isn't measuring opinion. It's manufacturing a citation.

Who paid, and what got released

Campaigns and advocacy groups poll constantly. Most of it never sees daylight. What gets released is the subset that helps, which means an internal poll is a selection effect with a topline attached. Use it as evidence that somebody wanted you to see a number, not as evidence of the number.

The tell isn't partisanship, it's disclosure. Serious operations publish the full questionnaire, the sample size, the field dates, the population, the mode and the weighting targets. Organizations that belong to the American Association for Public Opinion Research's Transparency Initiative commit to that in writing. When a release gives you a topline and nothing else, the absence is the finding.

The habit that actually protects you

One poll is an anecdote with decimals. Look at averages, look at trend lines, and weight the trend over the level. If six pollsters using six different methods all show movement in the same direction, something real is probably happening, even if none of them has the level exactly right.

And apply the whole checklist hardest to the poll you liked. That's not squishiness, it's self-defense. The poll that confirms what you already believed is the one most likely to slip past you unexamined, and the one most likely to leave you arguing from a number that fell apart on page 14.

Anybody can shout a topline. Reading the footnotes is the part that makes you dangerous in an argument.