How a comparison can rig the verdict before you read a word
Two columns, one side looks bad. Before you buy it, run five checks on how the comparison was built.

Quick answer: A comparison is only as honest as the two things being compared. Check the pairing, the denominator, the yardstick, the time window and the group mix. If any one is off, the verdict was baked in before you started reading.
The setup you've seen a hundred times
A story puts two things side by side. One party's voters and the other's. One city and another. One school's discipline record and its neighbor's. The headline tells you which column is the problem.
The numbers might all be real. That's what makes it work. Nobody lied, so nobody can be caught lying. The rigging happened earlier, in what got placed in the columns.
Every outlet does it, left and right, big and small. So the skill is the same whoever's doing it to you.
Check one: the pairing
The oldest trick is best against worst. Take the most polished representative of your side and the most unhinged one from theirs. Put them next to each other. Let the viewer finish the sentence.
Picture a segment that shows the loudest, angriest person at a rally on one side, then a calm, well-spoken person from the other. You'd never write "all of them are like this" on screen. You don't need to. The edit says it.
The fix is a question: is this person typical, or just available? Cameras find the extreme because the extreme is vivid. Ask what the average member of that group looks like, and whether the story ever tells you.
Run the same test on your own side's coverage. Right-leaning outlets reach for the most absurd campus protester the same way. Same trick, different costume.
Check two: count versus rate
Take made-up numbers, just to show the mechanics. City A had 90 burglaries last year and City B had 30. City A looks three times worse.
Now say City A has 900,000 residents and City B has 100,000. That's 10 burglaries per 100,000 people in A, and 30 per 100,000 in B. The "safer" city is actually the one where you're three times as likely to get burgled.
Raw counts reward the bigger group. Rates correct for size. When a story compares groups of different sizes and gives you only counts, it's either lazy or it's working. Look for "per 100,000," "per capita," "per game," "per dollar." If you can't find one, you've learned something about the story.
The reverse trap exists too. A tiny group's rate can swing wildly on a handful of cases. A rate of "double" means little if it went from two to four.
Check three: the yardstick
Say one outlet runs "Party X's scandal problem" and another runs "Party Y's scandal problem." Look at what each counted as a scandal.
If one list includes every resignation, every awkward tweet and every ethics complaint, and the other list includes only criminal charges, the lists aren't comparable. The longer one isn't proof of anything except a wider net.
The question to ask: would the other side's equivalent have made my list? If you can't tell what the definition was, the number can't carry the weight the headline puts on it. A fair comparison uses one definition for both columns and says what it is.
Check four: the time window
Pick a start date at a peak and nearly any line slopes down. Pick it at a trough and it climbs. Some starting points are natural, like a term of office or a law's effective date. Others are convenient.
When a chart or a claim says "since 2019" or "over the past year," ask why that date. Then ask what the picture looks like five years back. If the story gives no reason for its starting line, go find one that wasn't chosen by someone with a point to make.
This one cuts both ways, and politicians love it. Officials credit themselves for the rebound from a low point they inherited. Their opponents blame them for the dip that began before they took the job. A window that begins the day somebody took office isn't automatically honest.
Check five: who's in the group
This is the subtle one, and it fools smart people. It's called Simpson's paradox. A real example: in 1973, graduate admissions at the University of California, Berkeley, looked tilted against women. Overall, men were admitted at a higher rate than women.
Researchers took it apart department by department. Within most individual departments, women were admitted at rates similar to men's, and in some, slightly higher. The overall gap came from where people applied. More women applied to the most competitive departments, where nearly everyone gets turned away. More men applied to departments with high acceptance rates. The finding was published in Science in 1975.
Look at what that does to a headline. "Men admitted at higher rate" was true. "Women disadvantaged in the departments" was false. Both came from the same data. The first comparison lumped together groups that weren't alike.
Whenever you see an overall gap between two groups, ask whether they're made up of the same mix underneath. Age, region, job type, income and what each group was choosing to do can all hide inside one clean-looking number. Sometimes the gap survives the breakdown and means something real. Sometimes it disappears or flips. You don't know until somebody does the breakdown, and a story that won't tell you whether it did is asking for trust it hasn't earned.
The flip test
This is the check that takes ten seconds and no spreadsheet. Swap the names. Take the story's two sides and reverse them in your head. Same facts, same framing, same adjectives, opposite teams.
Would you accept it? Would you call it fair, or call it a hit job? If the reversed version makes you angry and the original doesn't, you've found your bias, not the outlet's. That's useful information, and it's free.
Then flip the coverage too. When one side's misstep gets ten stories this week, search whether a similar one on the other side got ten. A story can be accurate and still be one of the few that got told. What an outlet decides to cover, and how loudly, is a comparison in itself.
Opinion: check your own team first
This part's commentary, not news. Conservatives have good reasons to distrust much of the national press. Plenty of coverage over the years has treated our side as the problem by default, and noticing that is fair. But the reflex of reaching for the other team's flaws and waving through our own outlets' math is exactly how a reader gets played.
A person who only audits the other side isn't media literate. He's just picking a team. Real skepticism costs something. It means the chart that makes you feel good gets the same five checks as the chart that makes you angry.
None of this makes the reporter across the aisle a villain. Most people building these comparisons are rushed, working with whatever data they can find, and sometimes fooling themselves first. Be hard on the method. Stay decent about the person.
What to do tomorrow
The next time a story sets two things side by side, stop before you take the verdict. Ask five things:
- Are these two typical examples, or the most dramatic ones?
- Is it a count or a rate?
- Was the same definition used for both?
- Why does the timeline start where it does?
- Are the groups made of the same stuff underneath?
A story that survives all five has earned your belief. One that dodges them has told you what it is. For more on weighing what you read, try the four questions an editor asks before a story runs, and how to tell which outlets actually own their mistakes. And if the comparisons you see keep tilting the same direction, how your feed decides what you see may explain why.
Do it for a week, and you'll start to see the thumb on the scale. After that, it's hard to un-see.
Marcus Vale
Editor-in-Chief
Twenty years in magazines, most of it deciding what to cut. Writes about work, discipline and the decisions that compound.
Watch
Worth an hour of your evening
More from Media Literacy.
From channels we rate. Plays on YouTube.
Read next
