Why You Can’t Always Trust Medical Research Results

Clinical_research_questions

If you open the “Health” section of any online or paper publication, you’re bound to find plenty of stories saying “New study suggests X causes Y”, or something similar. The catch is usually an unusual correlation that grabs your attention.

The articles usually give the impression that there’s a linear relationship between the different variables being studied, and if you just ate more of some food, or did more of some behavior, then you’ll receive similar results.

If only it were that easy, right?

The problem is that new studies showing bizarre correlations are typically done using small sample sizes, and as more studies are done in larger numbers, the observed differences disappear and the original effect becomes random. However, the march towards randomness never makes the headlines.

Alex Tabarrok gave a great example today in Marginal Revolution. In 1992 an article published in the journal Science found that a gene variant of an enzyme called ACE significantly modified someone’s chances of having a heart attack. Woohoo! And, the article was published in Science, the journal of all journals.

It must be fact then, right?

As it turns out, not quite.

The study was done on 500 people, but as sample sizes grew larger the effects began to diffuse, until eventually there was no effect at all.

This phenomenon is not unique. Look at the graph below. Genetic variants of 8 different enzymes that had a statistically significant effect on diseases in small samples had no effects when performed on larger ones.

Natgen_funnel-plot_cropped

Now think about all the articles you’ve read stating “New study suggests pistacchios might cure Cancer” or some other vibe. Huh.

This effect is more robust when you consider that most research reports no significant results at all….but it doesn’t get published as often. No news is boring.

One report even suggested that the majority of published medical research is false. Sample sizes must be small, and Bayesian reasoning would predict that the majority of “findings” are actually false positives.

So the next time you read something about the latest trend, or finding that you can cure a disease with some food, take it with a grain of salt. Researchers are probably just thinking too small.

Turbocharge Your Body With 38 Different Superfoods

  • Raw, Unprocessed, and Non-GMO
  • Grass Juices That Alkalize Your Body
  • Medicinal Herbs That Revitalize Your Immune System
  • Adaptogens That Increase Your Energy
  • Probiotics That Aid Digestion And Detoxification
incredible greens

What Others Have To Say

---So What Are You Waiting For?---

About Jonathan Bechtel

Owner of Health Kismet, maker of Incredible Greens, a green superfood supplement that combines 35 different raw greens, herbs, probiotics, grasses and fruits into a sweet tasting powder.

Comments

  1. Interesting post, and very useful. I agree that reading the health news can be confusing if you take it all at face value. The concept of statistical significance has varying levels of usefulness, and a p-value is not the be-all end-all of research validity. In fact, in epidemiological research there’s a growing consensus that they should be avoided entirely.

  2. Jonathan Bechtel says:

    Al, I remember thinking in college that p-values were a way of window dressing otherwise flimsy research. In general, I also think it throws sand on the quality of a lot of the media you read w/ regards to diet, as they usually fling X causes Y stories all over the place w/ out a scintilla of awareness about the limitations of the research they’re talking about.

  3. Linda Amaral says:

    Example: Vioxx, which really worked well for me. The problem with human testing on new meds is that the group is too small. It’s harmfulness is not discovered until the med is released and thousands of people use it and start having disastrous side effects that forces the manufacturer to take it off the market and start paying damages to those who were harmed.Another example: Coffee used to a big bad thing. Now it turns out it’s a powerful antioxidant. Now they’re attacking salt. And I still say plastic is the probable culprit is so many odd diseases and syndromes that have appeared in the last 20 years, like fibromyalgia, autism, etc Just sayin’

  4. Linda Amaral says:

    Example: Vioxx, which really worked well for me. The problem with human testing on new meds is that the group is too small. It’s harmfulness is not discovered until the med is released and thousands of people use it and start having disastrous side effects that forces the manufacturer to take it off the market and start paying damages to those who were harmed.Another example: Coffee used to a big bad thing. Now it turns out it’s a powerful antioxidant. Now they’re attacking salt. And I still say plastic is the probable culprit is so many odd diseases and syndromes that have appeared in the last 20 years, like fibromyalgia, autism, etc Just sayin’

  5. This is so interesting to me and something I’ve independently thought of before, but haven’t seen anyone really write about it on a food/health blog. So my big question is – how do we (or I specifically) know what’s best for my body and what’s good? I like to think the milk thistle I’m taking is good for my liver, but what if the study I read about it is really just correlation and not causation? Should I just go on trial-and-error with my self, and ignore all the studies out there?

    • I attempted a blog that didn’t really go anywhere, but one of my entries talks about this – I’m not sure I’m incredibly well spoke/written, but you get the idea: http://tracyinthewild.blogspot.com/2012/05/correlation-vs-causation.html

    • Jonathan Bechtel says:

      Tracy, I know when I read research on something, I usually go on the assumption that one study all by itself doesn’t say much of anything. Ideally you’d want a number of studies across the three types of research: molecular/biochemical, which pin down the different mechanisms by which certain compounds behave in the body, clinical studies which test results in human patients, and epidemiological studies which study population wide effects of certain variables.

      Any one of these studies taken all by themselves have pretty serious issues. When you can combine all three of them (or get pretty close to it), then you’ve got something.

      This is why I don’t like news articles that begin with “New Study Shows That….”

Trackbacks

  1. […] says “new study suggests x might cause y” should not be trusted. See my post about the problems with interpreting medical research results for more information.2). The drug industry usually games studies to get things approved.3). The […]

  2. […] of times “important’ events that happen with small groups of people become entirely insignificant when conducted with large groups of people.Lots of scientific studies can’t have their results replicated by other research teams.When […]

  3. […] simply not a lot of formal evidence that maps a particular food to a particular condition.  I’ve written before how difficult it is to reproduce results in scientific research.  This is not limited to merely supplements and is probably more true for drugs.4). This chart […]

  4. […] Medical studies that study aggregates of complex phenomena can have limited value […]

  5. […] 1).  All the weight loss stuff was done on very small groups of people.  It amounts to “hey, take this stuff for a few weeks and see what happens.”  I’ve written before how these sorts of results are not useful. […]

Leave a Reply to Jonathan Bechtel Cancel reply