Understanding the Latest Research Findings: How to Be a Critical Interpreter of Health Information

By Elizabeth McDade-Montez, PhD | March 7, 2015
Senior Research Associate, ETR

We come across lots of health-related research findings reported in the news these days. Frankly, some of it is perplexing.

 

You may have heard the CDC’s recent recommendations that any young woman not on birth control should refrain from consuming alcohol. Perhaps you also saw some of the outraged reactions from social commentators.

 

Maybe you read about the classic psychology studies that weren’t replicated in recent research. Or the range of rumors flying around about Zika virus. And are you still hearing rumors online or from peers suggesting childhood vaccinations aren’t safe?

 

How does an informed reader sift through this constant stream of health information? When we are puzzled ourselves, how can health providers and educators support patients and clients trying to make sense of conflicting or suspect reports? What references can we trust when we endeavor to inform ourselves or support and guide others? 

Researchers Face Similar Challenges

I’m a health psychology researcher. My colleagues and I face this same challenge ourselves.

In any given year, dozens or even hundreds of new studies are likely to be published within our given field. Immediate access to the latest health research through popular and academic online outlets is both amazing and overwhelming. Trying to keep up with the latest research, interpret the findings and apply them to our own lives can feel like an impossible task.

Not only do we have access to an abundance of health information, but often times research reports present results that contradict one another. For example, you may have read about the toxic kale theory after hearing about kale’s super food powers. Or perhaps you heard that watching the popular MTV show 16 and Pregnant was associated with lower teen pregnancy rates, and then read the finding that watching 16 and Pregnant decreases teens’ perceived risk of pregnancy, which could lead to increases in teen pregnancy rates.

As informed professionals, we want to be able to draw conclusions from what we read and have a sense of how it might affect our patients and clients. We may even want to take steps to improve our own health based on what we’ve read.

With those goals in mind, what can we do to make sense of the latest health findings?

Four Basic Questions

Here are four basic questions I always ask myself when I come across a new research finding:

1. What is the original source of information?
2. What was the actual research?
3. How does this finding fit into the field?
4. Are there any issues or red flags in either the source or the research itself?

There are simple checks we can make on each of these questions to weigh the quality and relevance of the latest findings. Following these steps when we come across new research reports empowers us to be critical interpreters of health information and more powerful advocates for information that is sound and useful. These are also exactly the steps we want to explain to our clients to help them be critical consumers of all of this information.

1. What is the original source of information?

We receive health-related research findings from many outlets these days, including news, websites, social media and email, among others. Regardless of the outlet, quality reporting or information sharing should include a reference to an original source. Informed providers and consumers need to check the reliability and trustworthiness of original sources.

What to check

• The original source may be a scientific publication, an author/individual or a website.
• For individuals, examine the author’s credentials to assess whether or not the author has expertise in the area. A web search of the person’s name often leads to professional position and affiliation.
• For websites, consider whether the site is affiliated with a credible institution or organization (a university, medical school, government resource or respected national nonprofit, for example). Examining the organization behind the website will illuminate its credentials, as well as any potential biases (see question 4, below, for more on biases).

What about scientific publications?

The easiest-to-trust source of information is generally a scientific publication, which undergoes a peer review process before being published. This process includes having experts in the field read and comment on the study to determine whether the study quality is high enough for publication. However, even within scientific publications there are different levels of scientific scrutiny and publication rates.

For example, the prestigious Journal of the American Medical Association, or JAMA, rejects approximately 90.5% of all submissions. The average rejection rate across American Psychological Association journals in 2013 was lower at 76%. If you are curious about the quality of the journal, you can often find their rejection/publication rates on the journal’s website.

You can also look at that journal’s impact factor, which is a numeric indicator of how frequently other scientists reference studies published in that journal. A large impact factor means that journal has a greater reach. JAMA, for example, has an impact factor of 35, which is quite high. It’s rated 3^rd out of 153 journals in the category “Medicine, General & Internal.” On the other hand, the journal Transplantation Proceedings has an impact factor of .098, which is obviously considerably lower. (An easy way to check on a journal’s impact factor is simply to Google the journal name along with the phrase “impact factor.”)

Our imperfect world

In a perfect world, determining that the original source is a scientific publication should give us a sense of confidence in the findings. However, we don’t live in a perfect world.

One recent blemish in the field of scientific publications was uncovered in a sting operation of open access journals, which are growing in numbers. These online sources allow users to read articles for free, but often require a fee from the article’s authors in order for the paper to be published. Although many of these journals have a peer review process as rigorous as traditional journals, the sting operation revealed that 60% of targeted open access journals did not undergo peer review.

In addition, more than half of the journals targeted in this sting, set up by a science journalist, accepted a fake, fatally flawed scientific paper for publication. This finding adds a bit more complication to our trust in scientific publications, but it was also taken seriously by many publishers of open access journals. Some closed down the journals that failed to discern the fake paper.

One library scientist has published a list of “questionable, predatory” open access scientific publishers that you can find here. This operation reminded us all that peer review is essential to strong science.

2. What was the actual research?

The next step to making sense of the latest research findings is to take a good look at the actual study. Unfortunately, journalists and websites don’t always do the best job of summarizing or interpreting research findings, and headlines are often misleading.

The key lies in the description of the study from the original source, not a summary of the research reported in the news. Once you have access to an accurate description of the research, questions about Who, What, Where, When and Why can guide your critique of the actual study.

Who

Who refers to who was in the study. Were the researchers studying humans or animals? If it was human research, what types of people were in the study. Are they similar to you or your patients/clients? Is it a representative sample of participants, or a convenience sample, such as college freshmen selected only for their access and proximity?

How many people were in the study? You’ll want to determine if the study was conducted with a small group vs. hundreds or even thousands of participants. Generally speaking, the larger the sample size, the more robust and generalizable the findings will be.

What

What refers to the type of study design. I could write a whole blog post just on study designs, so I’m going to summarize and simplify the main types and issues here. One of the broadest ways to categorize research design is into experimental vs. nonexperimental.

Experimental designs are generally the most rigorous, especially those that are randomized, controlled trials. These studies are able (at times) to demonstrate cause and effect—if a provider offers intervention X, for example, the outcome for patients will be Y.

However, not all health topics can be studied experimentally. Nonexperimental designs include survey studies and observational studies. Survey studies can shed light on relationships among variables at one point in time (called a cross-sectional study) or over multiple points in time (called a longitudinal study, or a cohort study). But they can’t prove that one thing causes another and they can’t control for influences of unmeasured variables.

Observational studies (including case studies) include careful observations of behaviors or records as they occur in real life. These can describe how variables relate to one another in these settings. However, observational research cannot demonstrate cause and effect, and many variables of interest may not be captured in observations.

Another category of research design includes summaries of other studies, such as meta-analyses. These papers summarize results from multiple studies on a topic. They often conduct statistical analyses to determine the average relationship between variables or the average impact of one variable on another. These types of analyses often improve our understanding on health-related research questions (see, for example, a meta-analysis on the link between mindfulness-based stress reduction and health benefits).

Where and When

Where and When are pretty straightforward questions. Understanding where and when the research was conducted will provide insight into the applicability of the findings to your patients or clients today. Relationships between different health-related variables in one country or region of the world may vary relative to other parts of the globe.

For example, the prevalence of alcohol-related harm (e.g., alcohol dependence, alcohol-related disease) varies across countries not simply because of differences in how much alcohol is consumed, but also due to differences in culture, drinking context and wealth of the country (among other factors).

Recently published research (i.e., within the past 5 years), when available, is often more relevant than older research. If you are interested in how social media use affects youth today, for example, you’ll want to read research about today’s social media, rather than outdated social media such as MySpace (sorry, MySpace, no offense). In addition, many areas of health-related research move at a rapid pace; the latest research findings can make older research obsolete.

Why—and its relationship to conflict of interest

Why refers to the motivation for the research, which might not always be obvious. Generally speaking, researchers are motivated to find a cure, advance the field, create knowledge or seek social justice. Ideally, researchers behave in an objective manner in pursuing these goals.

However, all fields of science have the potential for bias at the individual and institutional levels. A conflict of interest occurs in “a situation in which financial or other personal considerations have the potential to compromise or bias professional judgment and objectivity.”

The most obvious example of a conflict of interest comes up in pharmaceutical research, where research funding comes from a pharmaceutical company that stands to profit from successful research on new drugs. However, conflicts of interest can occur in any research setting where there is the potential for financial or other personal gains.

Conflicts of interest are common and, in and of themselves, are not problematic. However, they are important for researchers to identify as they could lead to misconduct in research. Ideally, authors of scientific publications will disclose any conflicts of interest that they may have in conducting their research.

In addition, research involving humans should also be reviewed by a board (an institutional review board for the protection of human subjects) prior to commencing (and throughout a study) to assure that these potential conflicts are identified.

3. How does this finding fit into the general field?

This next question might be a bit more difficult to answer, but it’s important to keep in mind that science generally moves forward in small increments, with occasional big leaps forward. You may be able to determine from the original source what types of work this study is building upon and how it fits in with prior research.

Although less sexy than new, novel findings, replication studies are critical for building our knowledge. Novel findings are important to replicate in separate, unaffiliated laboratories to establish the robustness of the results. Although new, exciting findings are, well, new and exciting, confidence in the findings increases as new investigators are able to repeat and/or extend the work with new samples.

4. Are there any issues or red flags with the source or the study?

The last major consideration is whether there are any other issues or red flags that come up as you review the latest findings. Here are some possible issues that should have you questioning the information:

• You can’t identify an original source.
• There is a major commercial interest associated with the original source, and you can’t find an independent source to corroborate the research findings.
• The language comes across in a biased, non-neutral tone, or there are grammatical errors or misspellings throughout.

These issues may indicate that the source and/or findings are not credible, and that you should look elsewhere for more accurate information.

Time to Draw Conclusions

If you approach the latest headlines with these questions in mind, you’ll have an easier time interpreting results and drawing reasonable conclusions. Though this process may sound like a lot to do, it becomes fairly easy with practice.

And it’s necessary. As researchers and providers in the health care world, we have high standards to meet. There is a good deal of unreliable information out there, much of it looking similar to the well-grounded reports and studies. When we inform ourselves fully, we can provide better guidance to our patients and clients. We can also offer more informed and constructive responses when they bring up content—good or bad—that they have discovered in their own Internet searches.

 

Elizabeth McDade-Montez, PhD is a senior research associate at ETR. She can be reached at liz.mcdade-montez@etr.org.