Tag Archives: why science studies are usually wrong

Health science journalism–don’t just blame the journalists for getting it so wrong

Coffee is good for you, coffee is bad for you. Red wine will save your life, red wine will kill you. Low carb diet is the best, low carb diet is dangerous. Most major news outlets feature health science news with information designed to help us choose healthier lives. The problem is, much of it seems contradictory, headlines seem designed to scare us into attention rather than inform us. It’s ironic: the fear expressed by particularly the younger generation about food, pharmaceuticals, lifestyle choices and the like is matched only by their extreme distrust in the media. In other words, it seems the attention-desperate media have been successful in creating a state of fear, even while losing the public’s trust.

I must admit that I have blamed, unfairly it now seems, those attention-desperate reporters and editors. The Columbia Journalism Review has an outstanding analysis of health journalism and why the media most often gets the science news wrong. It’s titled “Survival of the Wrongest,” and in that awkward title is the premise which says that the process of getting science studies published in professional journals leads to bad studies. Studies that make good publication material but are fundamentally flawed. The reporters then pick up on these published studies and run with them compounding the wrong information.

It’s not a problem, says the article’s author David Freedman, of reporters doing sloppy reporting:  “Rather, science reporters—along with most everyone else—tend to confuse the findings of published science research with the closest thing we have to the truth. But as is widely acknowledged among scientists themselves, and especially within medical science, the findings of published studies are beset by a number of problems that tend to make them untrustworthy, or at least render them exaggerated or oversimplified.”

It is the studies themselves that are the problem. Freedman cites Stanford researcher John Ioannidis who, using several different techniques, has determined that “the overall wrongness rate in medicine’s top journals is about two thirds, and that estimate has been well-accepted in the medical field.”

In other words, just because a study is published in a major medical or science journal and has gone through all the peer review and editorial review processes, it does not mean the findings it presents are true. In fact, in two thirds of the cases, they are not.

That is astounding. But it raises a rather profound question. In all the trust studies I have seen it is the academics, the scientists, the experts who are trust above all others. This kind of information knocks the underpinnings out from the kind of trust. Just because a respected scientist says it is true, it ain’t necessarily so. Just because some outrageously expensive professional journal publishes the results, the conclusions are not necessarily right.

Well, I feel a bit better about the media–that is the general media. But, it hardly does much to answer the question “who do we believe? who can we trust?”

Does Freedman have an answer to that overarching question? “My advice: Look at the preponderance of evidence and apply common sense liberally.”

Great advice. Now, what are we to do about the state of fear that has been created by this nonsense?