How to Spot a Number That's Lying to You

Ten men are standing in a room. Nine of them make $50,000 a year. The tenth runs a hedge fund and clears $10 million. The average income in that room is over a million dollars.

Nobody lied. The math is right. And the number is useless, because not one man in that room lives anything like a million-dollar life.

That's the whole game. Most statistical deception in American media isn't fabrication. Fabrication gets caught. What gets through is a true number answering a question nobody asked, placed next to a headline that implies it answered a different one. You don't need a stats degree to catch it. You need about five questions and the willingness to be the guy who asks them.

Average of what, exactly

When somebody says "average," ask whether they mean the mean or the median. The mean is the total divided by the count. The median is the middle value, the one where half are above and half below.

On anything with a long tail, those two numbers live in different zip codes. Income. Home prices. Net worth. Medical bills. Time spent waiting for a contractor. The mean gets yanked upward by a handful of extreme values; the median shrugs them off. This is why the Census Bureau leads with median household income and why real estate reports usually give you median sale price. It's also why "average net worth of Americans in their 50s" headlines make normal people feel like failures. That number includes billionaires.

The mean isn't evil. If you're budgeting a company payroll, the mean is exactly what you want, because you're paying the total. If you want to know what a typical family experiences, the median is what you want. The tell is when someone picks the mean for a "typical person" claim. That's not a rounding choice. That's a decision.

Quick test: when a number about ordinary life feels higher than anybody you know, you're probably looking at a mean.

The denominator nobody printed

A raw count isn't a rate. This is the single most common trick in political coverage, and both parties run it with equal enthusiasm.

"Record number of X" is almost always true of something in a country that adds people every year. Record number of arrests, record number of crossings, record number of hospital visits, record dollars spent. Compared to what? Per capita? Per thousand residents? Per encounter? Adjusted for inflation? A federal budget line that hits an all-time high in nominal dollars may have shrunk in real terms, and you'd never know from the chart.

Ask out loud: per what?

You'll see the same move in safety arguments. Air travel looks dramatically safer than driving when you measure deaths per mile traveled, and considerably less dramatic when you measure per trip. Neither framing is fraudulent. They answer different questions, and the person choosing the denominator is choosing the answer.

Same with crime. A city with twice the population of another will naturally post more total incidents. If somebody hands you totals to compare two places, they've made the comparison for you and hidden their work. Go find the rate.

Percent of what, and percent of when

Two separate traps live inside the word "percent."

The first is relative versus absolute. "This doubles your risk" sounds alarming. If the risk went from 1 in 10,000 to 2 in 10,000, your absolute risk rose by one ten-thousandth. Both descriptions are accurate. Only one of them belongs in a headline. Whenever you see a percentage change in risk, go looking for the two raw numbers. If the article never gives them, that's information about the article.

The second is percent versus percentage point. If support moves from 40 percent to 44 percent, that's four percentage points, and it's also a 10 percent increase. Advocacy writing picks whichever version sounds bigger. Both are honest. Only one is clear.

And watch the base. Small starting numbers make spectacular percentages. A category that grows from 2 to 8 is "up 300 percent," which is technically a fact and practically a rounding error.

The subgroups can point the other way

In 1973, the University of California, Berkeley looked like it had a serious problem. Graduate admissions accepted roughly 44 percent of male applicants and about 35 percent of female applicants. Overall, a big gap.

Then a team of researchers, Bickel, Hammel and O'Connell, broke the data down department by department and published it in Science in 1975. Inside individual departments, women were admitted at rates that were mostly equal or slightly better. The aggregate gap came from where applications went: women applied in larger numbers to departments with low acceptance rates across the board, and men applied more to departments that admitted a higher share of everybody.

That's Simpson's paradox, and it's not a curiosity. It's a standing warning. A pattern in the pooled data can reverse when you look at the groups that make it up. Anybody can pick the level of aggregation that flatters their argument, and most people do it without even noticing.

So when a single top-line number carries an entire claim, ask what it looks like broken apart. And when a broken-apart number carries the claim, ask what the total looks like. The honest analyst shows you both.

Definitions are doing more work than the math

Before you argue about a number, find out what it counts.

"Gun deaths" is a real, sourced category from the CDC, and more than half of the American total in a typical year is suicide. That fact doesn't settle a single policy argument by itself, but it means two people citing the same true number can be describing two completely different problems. Same with unemployment: the headline rate the Bureau of Labor Statistics calls U-3 excludes people who've stopped looking, while its broader U-6 measure includes discouraged and part-time-for-economic-reasons workers. Both are published. Both are official. They're not the same number, and people quote whichever one fits.

Terms like "assault weapon," "living wage," "food insecure" and "extremist" all have specific operational definitions somewhere in the methodology, and those definitions decide the result before anybody runs the math. Find the definition. It's usually one PDF away.

Reading a poll without getting played

A few things make polling coverage immediately more readable.

• The margin of error applies to each candidate's number, not the gap. A three-point lead in a poll with a three-point margin of error isn't a lead. It's a tie with a suggestion.
• Crosstabs are noisy. A national poll of 1,000 people might have 80 respondents in the subgroup a headline is built on. The error bar on that slice is enormous.
• Registered voters and likely voters are different populations and produce different results in the same survey.
• Question wording moves answers, sometimes by double digits. Read the actual question. It's in the toplines.
• One poll is an anecdote. Averages of many polls are far more useful than the single one that made news, and the one that made news is usually the outlier. That's why it made news.

The sixty-second version

Next time a number stops your thumb, run these before you share it.

1. Mean or median?
2. Per what? Where's the denominator?
3. Relative change or absolute change, and what were the raw counts?
4. What happens when you split it into groups?
5. Who defined the category, and where's the methodology?

You'll kill about half the numbers you were about to repeat. Some of them will be numbers you badly wanted to be true, and that's the part that stings.

Do it anyway. A man who only audits the other side's data isn't skeptical. He's just shopping.