Every number answers a question nobody printed
A statistic can be accurate and still mislead you. Four questions will tell you which one you're looking at.

Quick answer: A number is only as honest as its comparison. Before you repeat one, ask out of what, compared to when, compared to whom, and counted how.
Say a headline claims a common medication "doubles your risk" of a rare condition. That sounds alarming. Now say the risk went from 1 in 10,000 to 2 in 10,000. (Those figures are invented for the illustration, and that's the point.) Nothing in the headline tells you which version you're reading. In the second one, your odds of being fine are still 99.98 percent.
Every statistic is the answer to a question. The question usually isn't printed. Your job is to find it.
Out of what?
Start with the denominator, the number underneath the fraction.
Raw counts reward size. The biggest states have the most of nearly everything: the most car accidents, the most bakeries, the most arrests. A headline that says a big state "leads the nation" in something may only be telling you it has a lot of people. Ask for the rate, meaning per 100,000 people or per mile driven or per thousand workers.
The same trap shows up in relative change. "Up 50 percent" from two cases is three cases. "Down 40 percent" from a number that had spiked is just a return to normal. When you see a percentage, go find the number it was a percentage of.
And watch the difference between percent and percentage points. If a rate goes from 4 percent to 6 percent, that's two percentage points. It's also a 50 percent increase. Both are true. Whoever wants you worried will use the second one.
Compared to when?
Every trend has a start date, and the person choosing it often knows what the line will look like.
In 2020, the U.S. murder rate jumped by roughly 30 percent, the biggest one-year rise in modern records. In the years since, it has fallen sharply. Start the clock at 2019 and stop at 2021, and you can say violence exploded. Start at 2021 and stop at 2024, and you can say it collapsed. Both people can cite the FBI. Neither is lying. Only one of them is helping you.
Jobs numbers work the same way. More than 20 million jobs vanished in the spring of 2020. Any president whose term began after that could claim record job growth, and the claim would be arithmetically true, because much of it was people going back to work they'd lost. A president whose term ends in a crash gets the opposite treatment. Both parties have used this move whenever the calendar favored them.
The fix is simple. Ask what the line looked like for ten years before the start date. If the speaker's chart begins at a peak or a bottom, you've found the angle.
Compared to whom?
A comparison only means something if the two groups are alike in the ways that matter.
Take the pay gap. The widely quoted figure, women earning somewhere in the low-to-mid 80s in cents on a man's dollar, compares median full-time earnings across everybody. That lumps together different occupations, different hours and different years of experience. When researchers adjust for those factors, the gap shrinks. How much is left, and why, is argued over by serious people.
So "women get 82 cents for the same work" overreaches, because the raw number doesn't measure the same work. And "there's no gap at all" overreaches in the other direction, because the adjusted gap generally doesn't hit zero. Each side is holding half a number. Your move is to ask what was controlled for.
Counted how?
Definitions do quiet work.
The unemployment rate in the headline counts only people who looked for work in the last four weeks. A broader measure, known as U-6, also counts people who gave up looking and part-timers who want full-time hours. It's always higher. Which one gets quoted tends to depend on who holds the White House.
"Gun deaths" is another one. In recent years, more than half of U.S. gun deaths have been suicides. That's a real figure, but the headline phrase makes most readers picture crime. Going the other way, "most of it is suicide, so set it aside" waves off a real loss of life. The honest move is to split the number and look at each piece.
Estimates of how often people use a gun in self-defense run from tens of thousands a year to millions, depending on the survey and how it words the question. When one source says one thing and another says a hundred times that, you're not looking at a fact. You're looking at a method dispute. Read the method.
Then there's the word "average." Picture ten people at a bar earning ordinary wages. A billionaire walks in. The average income in the room just shot up, and nobody got a raise. The median, the middle person, barely moved. Whenever you see an average of something lopsided, like income, home prices or net worth, ask for the median.
Read the chart, not just the headline
Look at the vertical axis first. A bar chart that starts at 90 instead of zero can turn a two-point difference into a cliff. Truncated axes aren't always dishonest, since a line chart of temperature doesn't need to start at zero. But if the bars are the story, they should start at zero.
Then look at who made it. A chart from an advocacy group, a campaign, a union, a trade association or a newsroom can all be accurate and all be selected. Selection is the part you can't see on the page.
Two minutes, five checks
You don't need a statistics degree. Run any number that makes you feel something through this:
- Find the denominator. Is it a rate or a raw count?
- Move the dates. Would the story change if the chart started five years earlier?
- Check the comparison. Are the two groups really alike?
- Find the definition. What exactly got counted, and what got left out?
- Look for the primary source. Not the article about the study. The study, or the agency table, even if you only read the summary.
If a piece won't tell you any of this, that tells you something too. Editors ask similar things before a story runs, as in the four questions an editor asks before a story runs. It's also worth noticing which outlets fix a bad figure when they get caught, which How to Tell Which Outlets Actually Own Their Mistakes covers.
None of this is cynicism. Most statistics are honest, and most people repeating them are sincere. Context is just the part that rarely fits in a headline or a tweet.
If you've got kids, this is a skill worth passing down early. Teach Your Kid to Check It Before They Believe It is a good place to begin.
Now the hard part. Run these checks on the numbers you like first. The ones that confirm what you already believe are the ones you'll never question on your own, and they're exactly where this habit pays off.
Marcus Vale
Editor-in-Chief
Twenty years in magazines, most of it deciding what to cut. Writes about work, discipline and the decisions that compound.
Watch
Worth an hour of your evening
More from Media Literacy.
From channels we rate. Plays on YouTube.
Read next
