How To Evaluate Scientific Research
I received an email from one of our customers with a great question,
“Thank you for reminding me to be more cautious about what I believe that I see on the internet. But how am I or anyone who doesn't have a science background supposed to know what to look for in all of those studies we read about?”
The science teacher in me just loved this question! So, I decided to deviate from my original blog plan about natural deodorant to try and help answer the question.
Every day we are bombarded with information about science from newspapers, radio, television, and the internet. As consumers, we often need to evaluate scientific claims in advertising and the media.
Advertising is used to sell products and scientific claims are often used in advertising to convince us of the benefits of those products. Making sense of it all and sorting out credible claims and credible scientific studies can be very difficult for the non-scientist. So how do we, as consumers, learn what is good science and bad science?
A true evaluation of a piece of scientific research requires a thorough understanding of research methods as well as the topic being researched.
The purpose of this blog is to provide some basic information on how to look with a more critical eye at scientific claims and research.
To begin with I am always skeptical of “Headlines." Not only are they usually misleading, but they are also usually inaccurate—remember they are trying to get your attention. For example, the headline reads, “Strawberries Cure Cancer.”
But when you read the article or research paper, the study actually shows there was a correlation between rats that ate one pound of strawberries per day and a reduced risk of cancer.
First of all, humans are not rats and since the average rat only weighs about 1/2 pound, I can not even imagine how many pounds of strawberries the average person would have to eat in order to repeat the study in humans. Also, be careful of headlines that include words such as "a link has been found" between two things, like "a link has been found between eating strawberries and cancer." Words like "link" too often mean that the results being reported are not statistically significant.
A lot of people use the words, “studies have shown….,” without providing a citation to the study. I know that I am guilty of that myself. Right or wrong, I guess I don't really question a study that says that "cocoa butter is moisturizing.” But when it comes to using a study to make important decisions in my life, I become a skeptical consumer—the information is false until proven true.
I like to see a citation so that I can read the original study. Although I have a good science research background, I may still find it difficult to read a piece of scientific research, so I start by asking a few simple questions about the study:
- Who are the authors or researchers? What is their expertise?
- Are the authors affiliated with a reputable institution?
- Is the study published in a reputable journal?
- Who paid for the study? Although you may not be able to find this information, I get a bit concerned when a study is paid for by the company trying to sell me the product.
- Have any other researchers commented on or repeated the study? In other words, have the findings been checked by other experts?
Scientific inquiry, whether it is a student creating a science fair project or someone doing scholarly research, follows some form of the same basic process called the “Scientific Method” (shown in the graphic to the right from eosmith.org).
It may seem complicated, but we actually use the steps in this process every single day of our lives to make simple decisions. For example, the alarm clock rings and you get out of bed . . .
This is a very simplistic example of the scientific method, but it shows the flow of how research is conducted and written. A scientific research paper usually contains the following sections:
Abstract: This first section is actually the last piece written. It is a complete summary of the paper. It describes the purpose of the paper, how the work was carried out, the results and the conclusions.
Introduction: This section gives background information about the topic of the paper and answers questions like, What is the big question—what problem is being studied--what is the purpose of the study? Why is it an important question? What information was already known about the question before the study? What do the authors hope to learn from this research? What is the hypothesis?
Materials and Methods: The researchers describe how they designed the experiment or research to test their hypothesis. This section provides technical details and answers the questions: How did the researchers solve the problem--how did they carry out the experiment? This section also serves as an instruction manual for those who want to repeat the experiment. This is one of the most important but difficult sections to understand. See some basic questions to look for below.*
Results: This is an objective summary of the data and observations obtained in the study. It is usually full of formulas and statistics and can also be difficult to read. Take a look at the graphs and charts to get an idea of the results. No interpretation of the data or conclusions about what the data might mean is given in this section.
Discussion: The authors give you their opinions, draw conclusions about the results, and interpret the data in relation to the original problem or hypothesis. It is also the place where the researchers should evaluate their own experiment or study. Does the author identify any weaknesses in their own study? Is more research needed? The discussion tells the author's opinions which are not necessarily facts.
My goal when reading a scientific paper is to identify the claim the author is making and to see how well the research supports that claim. Reading the abstract is a good place to start for a beginner. But remember, since the abstract includes a summary of the whole research project, it also includes the opinions of the researcher. Don't let their bias persuade you! I like to start with the introduction. In general, the easiest sections for a non-expert to understand are the introduction and the discussion.
When scientists do research the most important part is setting up a well-constructed experiment in order to test their hypothesis.
It is the experimental procedure, the materials and methods, that really determines the validity of a study.
If the experiment is not well constructed, the study is meaningless. This section is often very difficult to read and understand.
So how do we determine if the experiment was well designed? *Here are a few basic questions to ask:
Were there enough people in the study to really show any statistical significance?
For example, two studies that found high levels of aluminum in women with breast cancer were carried out with less than 20 women in each study. It is difficult to draw reliable conclusions from 20 people. While it does not always mean that the findings are invalid, it does mean that further testing in more people is needed to make sure.
Was there a control group?
A control group in an experiment is the group that does not receive any treatment or does not have the disease or problem being researched. It serves as a point of comparison.
For example, a study published in 2003 was based on responses from questionnaires sent out to women who had breast cancer. The researcher reported that women who were diagnosed with breast cancer at a younger age said they used antiperspirant earlier than women who were diagnosed when they were older. But the study design did not include a control group of women who used antiperspirants at an early age and did not have breast cancer.
What about the Variables?
When teaching my students how to construct a good experiment I would begin with a simple exercise. The question they were required to answer was “how long does it take to melt an ice cube?” They were given no more information and not allowed to ask me any questions. The following day I would go around the room and write their answers on the board. The time to melt an ice cube varied from under one minute to 2 ½ hours.
The kids were quick to tell me that this was a bad experiment. The ice cubes were different sizes; I did not tell them how to melt the ice cube (a couple students actually used a microwave); some placed their ice cube on a cold window sill; etc. More importantly, they did not understand the purpose--what was the real question I was trying to answer.
What they learned from this silly experiment is that if we don’t know the purpose of the experiment and if we don't control the things that can affect the outcome of our experiments, what scientists call the “controlled variables,” we will never get accurate results.
I then asked, what if we change the question to “how long does it take to melt a one-ounce ice cube at room temperature of 70 degrees F?”
The students quickly pointed out that although my new question was more specific, it was still flawed. Different shapes of ice cubes have different surface areas; what if there is a fan overhead; what if the ice cube sat in the sunlight but the room temperature was still 70; etc. What they were learning was how difficult it is to control variables even in a very simple experiment.
Now imagine trying to control the variables when doing an experiment using human beings. For example, researchers have found a relationship between the development of breast cancer and certain factors such as early age at the first period; fewer children; older age at first childbirth; hormone replacement therapy; and obesity. When reading a study about the effect of antiperspirants on breast cancer we must ask, "Was there an attempt to control for these or other variables?"
Do the correlations always reflect "cause and effect?"
Scientists often look for cause and effect relationships in nature. This is called a correlation, or a
measurement of the relationship between two things.
We must be careful not to assume that because two things are happening at the same time, they exist in a cause and effect relationship with each other.
For example, in the graph above you can see that as per capita (per person) cheese consumption increased so did deaths by bedsheets. So this graph shows that there is a correlation between the amount of cheese we eat per person and the number of people who die by becoming tangled in their bedsheets.
While it is quite an interesting correlation, it is probably safe to assume that eating more cheese did not actually cause more deaths by bedsheets.
This may help explain some of the conclusions the researchers found.
For example, there was a researcher that reported that women who were diagnosed with breast cancer at a younger age said they used antiperspirant and started shaving their underarms earlier. Since underarm odor and the development of underarm hair comes with puberty, it is probable that women who started using antiperspirants or shaving earlier in life went through puberty earlier which means they had their first menstrual period earlier. Starting periods early is already considered a risk factor for breast cancer, so was the increase in breast cancer really due to the early use of antiperspirants or the early onset of menstruation?
To make a somewhat simplistic analogy I can relate this topic to the process we go through when buying a new car. Most of us are not experts on cars and we know that it is not a good idea to rely solely on the information provided by the car salesman. So we do as much research as possible, ask good questions and then do more research in order to make the best choice we can.
Looking for understanding beyond the science jargon is no easy task. But to become educated consumers we must try to see beyond the headlines, take a good look at the source of the information, gather as much information as we can and ask lots of questions in order to make informed choices in our lives.
This blog was in no way intended to be a comprehensive lesson on evaluating research. Students going into fields of science or scientific research often take an entire semester course, or more, just to learn how to read, interpret, and evaluate a piece of scientific research. I hope this information was helpful.
Stay tuned for the last part of the series: Antiperspirants & Breast Cancer Part 4: Natural Deodorants