Why you can't trust journalism

Everyone wants one thing: your clicks (and retweets, and likes, and citations). Most writers sincerely want the truth too. Sadly the two are not always compatible. – Chris Blattman

If you’re a public-health nerd, you’re almost certainly aware of the Worm Wars. If you’re a media nerd, you’re almost certainly aware of the Nail Wars. If you haven’t heard of either, that’s fine. But if you compare the two wars, which are similar in many ways, it’s hard to come away feeling good about journalism.

Don’t worry: I’m not going to talk about deworming, and I’m not going to talk about the labor practices of New York City nail salons. But these controversies present a great opportunity to look at how we discover what is true. The lesson, if you compare the two wars with each other, is that the truth, insofar as it even exists, is always hard to find – but that it’s particularly hard to find in journalism.

The most impressive achievement of any publication, in science or journalism, is that it changes people’s minds on a certain subject. And both the Worm Wars and the Nail Wars are based around groundbreaking blockbusters which did just that. In the case of the Worm Wars, the publication in question was a paper by Edward Miguel and Michael Kremer about intestinal helminths; in the case of the Nail Wars, it was an investigation by Sarah Maslin Nir into nail salons. Both were published by highly prestigious publications: Econometrica and the New York Times, respectively. And both have recently been called into question. The International Journal of Epidemiology published a paper calling the deworming study into question; the New York Review of Books published an article calling the NYT story “demonstrably misleading”. Cue Wars.

The best article about the Worm Wars came from the great science journalist Ben Goldacre, who immediately saw the bigger picture. The original paper from Miguel and Kramer was – is – “an excellent piece of research”, he says. But all research benefits from close study and painstaking attempts to replicate findings. And reanalyzed research is almost impossible to find, in any field.

That makes little sense, on its face. If you reanalyze an existing dataset, the cost of doing so is a tiny fraction of the cost of the original research, while the scientific value of doing so is almost as great as that created by the original research.

There’s an obvious media analogy here. In media, there are people doing original journalism, and then there are lots of other people who are curating and contextualizing and aggregating and linking and rewriting and explaining and generally doing stuff which is parasitical on the original journalism. Both are important, although some people think that there are too many of the latter genus of journalists, compared to not enough of the former.

Well, that’s not a problem in science. In science, everybody seems to be doing original research, and there aren’t nearly enough people going back and re-examining that original research, or trying to replicate it, or anything like that.

Goldacre’s great mission in life is to get much more original research out into the open, to enable much more of the kind of reanalyzing and remixing that we now take for granted in the journalism world. It’s a noble mission, and he’s finding real success. If you’re a scientist who refuses to open up your dataset and your detailed methodology, people increasingly start looking at you funny, because that attitude is deeply at odds with the scientific method. If you want to find the truth, you need multiple people all looking critically at the same data, and having an open and transparent debate about how to interpret it.

But here’s the thing: it turns out that the scientific world is actually far, far ahead of the journalistic world on these matters. Yes, the world of online journalism is full of parasites, and a lot of those parasites have real value. But all that the parasites have to go on are published articles: no one is transparent about the process that created those articles. No one shows their work, and no one ever tries to replicate anything.

Shortly after I started working at Condé Nast Portfolio, they sent me on a media-training course, and the main thing I learned at that course was what not to say. If I was being interviewed about a story we had published at Portfolio, a story which, presumably, I would have written, then my answers could only reiterate the facts in the piece. Anything which had been taken out of the piece, anything I had learned along the way but which had not actually been published – anything which might make the story even the tiniest bit more complicated than the final edited article – about all of that I was to be careful to say absolutely nothing at all. (This is why, if you read an article and then you see the journalist interviewed on TV, you’ll almost never learn anything new.)

Not all journalists follow these edicts (I don’t, and never did), but the fact that they exist at all tells you a great deal about how journalism works. The minute that anybody starts asking critical questions about any major piece of journalism, the author and the editor(s) tend to go very, very quiet. The most common outcome is the most depressing: a pro-forma and utterly unhelpful response from the publication’s PR department, saying “we stand by our story”.

This passage, for instance, from Richard Bernstein’s NYRB attack on the NYT, surprises only insofar as Bernstein got any response from NYT journalists at all:

I was genuinely mystified by this matter of the classified ads, and I wanted to see if there was some explanation for them. And so, two days after part one of the Times exposé appeared, I emailed several senior Times editors, including Mr. Baquet, as well as Margaret Sullivan, the Times’s public editor, who represents readers’ interests vis-a-vis the editors, pointing out what appeared to be the paper’s misrepresentation of the ads. I received cordial replies from editors, but my questions about the ads were ignored, except by Ms. Sullivan.

Margaret Sullivan, of course, does not represent the NYT. She is well placed to get answers from editors, but if you’re not Margaret Sullivan, forget it. It’s not going to happen.

Big journalistic stories always have many layers of editors and lawyers involved. And while at some level, in principle, those people are interested in telling the truth about the world, in practice, they are much more interested in making sure that any given statement is factually and legally watertight. Beyond that, they want something big, something punchy, something powerful. They want a narrative, with good guys and bad guys. And, of course, they want their story to be shared, and to elicit government investigations, and to win awards. (Those awards, it’s worth noting, are given out by juries, but the juries generally have no real tools with which they can critically judge the story in question. Journalism-award juries almost never take it upon themselves to re-report the stories they’re judging, or to go back to the editors and ask questions about them.)

Journalism can be superficial or it can be deep, but it’s nearly always going to look rushed and hurried in comparison to peer-reviewed empirical science. And yet, as Goldacre has reminded us many times, even peer-reviewed empirical science is often wrong. (As John Ioannidis famously demonstrated, “Most Published Research Findings Are False.”) If a scientific paper’s findings are surprising, there’s a very, very good chance that it’s wrong: extraordinary claims require extraordinary evidence. This is a known problem, with a known solution – and the solution is to check, to revisit, to reanalyze.

And then there’s journalism, where reporters come out with shocking and surprising stories every day, and no one ever gets to reanalyze the underlying reporting. Think of all the people who end up not being quoted at all, or who are only quoted anonymously, and remember that in journalism, what you leave out can be much more important than what you put in. (That’s Bernstein’s main charge against the NYT’s nail-salon story: that the paper was highly selective about what it published, and that while certain things in the story might have been narrowly true, the story as a whole was not the whole truth.) Journalists know full well how deeply wrong journalism can be and often is, but we try to put that out of our minds. There’s even a name for that self-deception: the Gell-Mann Amnesia effect.

It’s entirely possible that the NYT’s story was spot-on. The problem is that we have nothing to check it against. Woe betide any journalist who pitches an editor on a story examining wage conditions in New York’s nail salons: you’d be laughed out of the room immediately. Now that the NYT has published its version of the story, the story has Been Done. The rest of the media doesn’t look at it and think “wow, there’s a subject we should examine ourselves”. Rather, they think the exact opposite: “there’s a subject we must Never Touch”.

Journalism would be greatly improved were it to learn a lesson from empirical science, which is that even the best and most assiduous work can turn out on examination to be deeply flawed. That doesn’t mean the original piece was sloppy, it just means that everybody is human, and that the truth is usually messy, and hard to find, and even harder to encapsulate in a neat story with a compelling narrative. Whenever there’s only one story on any given subject, then relying on that one story is always going to be dangerous. And yet, the way the journalism world works, once there’s one big compelling story, no one else tends to ever go near that area again.

The result is unedifying war. On the one side you have the reporter and editors who worked on the original story, and who have every incentive to defend it as being exhaustive and comprehensive and utterly correct in all particulars. On the other side you have people who, in the absence of any extant counternarratives, are forced to start attacking the original story as being full of journalistic malfeasance. Both sides claim that the other side is being highly selective and disingenuous in terms of choosing facts and making arguments, and things rapidly become personal, with sides being picked and enemies being made. At no point does everybody agree to work together to help reveal a bigger truth; rather, everything descends into a shooting war, from which no one emerges unscathed.

At some level, everybody knows that this state of affairs is suboptimal. The problem is that there’s no Ben Goldacre of journalism, no one urging cooperation and replication in the service of a greater good. Science is a great enterprise, mostly run by governments and not-for-profit universities, where everybody in a sense has the same job. Scientific rivalries can be vicious, to be sure, but virtually all scientists at least pay lip service to an ideal whereby all of science is working collaboratively to deepen our knowledge of the world. The cut-throat for-profit world of journalism, by contrast, is much more antagonistic. What’s more, journalism is also infested with lawyers, who always give the same advice: say nothing that hasn’t been carefully vetted; be as unhelpful as possible. That advice is great if you want to avoid a lawsuit. But it’s horrible for anybody who’s simply interested in finding out what’s true.

If we’ve learned anything from science, it’s that gathering knowledge is a complex, iterative task involving thousands of different people. We think we know things, we test them, we refine what we know, we disappear down wrong paths, we turn around and go back, and we slowly end up knowing more and more over time. Thanks to that process, even the Worm Wars have been reasonably civil, at least by journalism-war standards: no one’s accusing anybody else of being a disgrace to their profession. And the analyses have been very well-informed.

Sadly, that’s not how journalism works. In journalism, you have a big hit or you don’t, and either way everybody generally assumes that you’re right unless someone says that you’re wrong, in which case there’s a fight and somebody’s reputation is likely to end up severely damaged. What I’d love to see would be a nonprofit journalism outfit which did nothing but re-report any big or interesting scoops. The idea would not be antagonistic: rather, the best outcome would be a true positive, which confirmed and underscored the original finding. But I’m not holding my breath.

Journalists are, at heart, storytellers more than they are empiricists. If you think that upon reading a single article, or watching a single documentary, you are magically in possession of The Truth – about nail salons or anything else – then you are deluding yourself. You’re only getting a single problematic sliver of the truth. And, sadly, that single problematic sliver is likely to be all you’re ever going to get.