How To Evaluate Scientific Research
After writing three blogs about the controversy surrounding antiperspirants as a cause for breast cancer, I received an email from one of our customers with a great question.
“Thank you for reminding me to be more cautious about what I believe that I see on the internet. But how am I or anyone who doesn't have a science background supposed to know what to look for in all of those studies we read about?”
The science teacher in me just loved this question! So, I decided to deviate from my original blog plan about natural deodorant to try and help answer the question.
Every day we are bombarded with information about science from newspapers, radio, television, and the internet.
As consumers, we often need to evaluate scientific claims in advertising and the media.
Advertising is used to sell products and scientific claims are often used in advertising to convince us of the benefits of those products.
Making sense of it all and sorting out credible claims and credible scientific studies can be very difficult for the non-scientist. So how do we, as consumers, learn what is good science and bad science?
A true evaluation of a piece of scientific research requires a thorough understanding of research methods as well as the topic being researched.
The purpose of this blog is to provide some basic information on how to look with a more critical eye at scientific claims and research.
Headlines Are Often Misleading
To begin with I am always skeptical of “Headlines." Not only are they usually misleading, but they are also usually inaccurate—remember they are trying to get your attention. For example, the headline reads, “Strawberries Cure Cancer.”
But when you read the article or research paper, the study actually shows there was a correlation between rats that ate one pound of strawberries per day and a reduced risk of cancer.
First of all, humans are not rats and since the average rat only weighs about 1/2 pound, I can not even imagine how many pounds of strawberries the average person would have to eat in order to repeat the study in humans.
Also, be careful of headlines that include words such as "a link has been found" between two things, like "a link has been found between eating strawberries and cancer." Words like "link" too often mean that the results being reported are not statistically significant.
A lot of people use the words, “studies have shown….,” without providing a citation to the study. I know that I am guilty of that myself. Right or wrong, I guess I don't really question a study that says that "cocoa butter is moisturizing.” But when it comes to using a study to make important decisions in my life, I become a skeptical consumer—the information is false until proven true.
Find the Original Research
I like to see a citation so that I can read the original study. Although I have a good science research background, I may still find it difficult to read a piece of scientific research, so I start by asking a few simple questions about the study:
- Who are the authors or researchers? What is their expertise?
- Are the authors affiliated with a reputable institution?
- Is the study published in a reputable journal?
- Who paid for the study? Although you may not be able to find this information, I get a bit concerned when a study is paid for by the company trying to sell me the product.
- Have any other researchers commented on or repeated the study? In other words, have the findings been checked by other experts?
Understand The Scientific Method
Scientific inquiry, whether it is a student creating a science fair project or someone doing scholarly research, follows some form of the same basic process called the “Scientific Method.”
It may seem complicated, but we actually use the steps in this process every single day of our lives to make simple decisions. For example, the alarm clock rings and you get out of bed . . .
This is a very simplistic example of the scientific method, but it shows the flow of how research is conducted and written.
A Scientific Research Paper Usually Contains the Following Sections
Abstract: This first section is actually the last piece written. It is a complete summary of the paper. It describes the purpose of the paper, how the work was carried out, the results, and the conclusions.
Introduction: This section gives background information about the topic of the paper and answers questions like, What is the big question—what problem is being studied--what is the purpose of the study? Why is it an important question? What information was already known about the question before the study? What do the authors hope to learn from this research? What is the hypothesis?
Materials and Methods: The researchers describe how they designed the experiment or research to test their hypothesis.
This section provides technical details and answers the questions: How did the researchers solve the problem--how did they carry out the experiment?
This section also serves as an instruction manual for those who want to repeat the experiment. This is one of the most important but difficult sections to understand. See some basic questions to look for below.*
Results: This is an objective summary of the data and observations obtained in the study. It is usually full of formulas and statistics and can also be difficult to read. Take a look at the graphs and charts to get an idea of the results. No interpretation of the data or conclusions about what the data might mean is given in this section.
Discussion: The authors give you their opinions, draw conclusions about the results, and interpret the data in relation to the original problem or hypothesis. It is also the place where the researchers should evaluate their own experiment or study. Does the author identify any weaknesses in their own study? Is more research needed? The discussion tells the author's opinions which are not necessarily facts.
How To Read Scientific Research
My goal when reading a scientific paper is to identify the claim the author is making and to see how well the research supports that claim. Reading the abstract is a good place to start for a beginner. But remember, since the abstract includes a summary of the whole research project, it also includes the opinions of the researcher. Don't let their bias persuade you! I like to start with the introduction. In general, the easiest sections for a non-expert to understand are the introduction and the discussion.
When scientists do research the most important part is setting up a well-constructed experiment in order to test their hypothesis.
It is the experimental procedure, the materials and methods, that really determines the validity of a study. If the experiment is not well constructed, the study is meaningless. This section is often very difficult to read and understand.
So how do we determine if the experiment was well designed? *Here are a few basic questions to ask:
Were there enough people in the study to really show any statistical significance?
For example, two studies that found high levels of aluminum in women with breast cancer were carried out with less than 20 women in each study. It is difficult to draw reliable conclusions from 20 people. While it does not always mean that the findings are invalid, it does mean that further testing in more people is needed to make sure.
Was there a control group?
A control group in an experiment is the group that does not receive any treatment or does not have the disease or problem being researched.
It serves as a point of comparison. Without the control group, there would be nothing to compare the treatment group to.
For example, a study published in 2003 was based on responses from questionnaires sent out to women who had breast cancer. The researcher reported that women who were diagnosed with breast cancer at a younger age said they used antiperspirants earlier than women who were diagnosed when they were older. But the study design did not include a control group of women who used antiperspirants at an early age and did not have breast cancer.
What about the Variables?
When conducting an experiment, the independent variable is the condition or factor you manipulate or change. The dependent variable is the outcome of the manipulation. For example, if you are measuring how the amount of sunlight affects the growth of a type of plant, the independent variable is the amount of sunlight. You can control how much sunlight each plant gets. The growth is the dependent variable. It is the effect of the amount of sunlight.
A controlled variable is any factor that is held constant during an experiment.
A single experiment may have many control variables which are very important because they could affect the outcome.
When teaching my students how to construct a good experiment I would begin with a simple exercise. The question they were required to answer was “how long does it take to melt an ice cube?” They were given no more information and were not allowed to ask me any questions. The following day I would go around the room and write their answers on the board. The time to melt an ice cube varied from under one minute to 2 ½ hours.
The kids were quick to tell me that this was a bad experiment. The ice cubes were different sizes; I didn't tell them how to melt the ice cubes (a couple of students actually used a microwave); some placed their ice cubes on a cold window sill; etc. More importantly, they did not understand the purpose--what was the real question I was trying to answer.
What they learned from this silly experiment is that if we don’t know the purpose of the experiment and if we don't control the things that can affect the outcome of our experiments, what scientists call the “controlled variables,” we will never get accurate results.
I then asked, what if we change the question to “how long does it take to melt a one-ounce ice cube at room temperature of 70 degrees F?”
The students quickly pointed out that although my new question was more specific, it was still flawed. Different shapes of ice cubes have different surface areas; what if there is a fan overhead; what if the ice cube sat in the sunlight but the room temperature was still 70; etc? What they were learning was how difficult it is to control variables even in a very simple experiment.
Now imagine trying to control the variables when doing an experiment using human beings. For example, researchers have found a relationship between the development of breast cancer and certain factors such as early age at the first period; fewer children; older age at first childbirth; hormone replacement therapy; and obesity. When reading a study about the effect of antiperspirants on breast cancer we must ask, "Was there an attempt to control for these or other variables?"
Do the correlations always reflect "cause and effect?"
Scientists often look for cause-and-effect relationships in nature in order to identify the extent to which one variable relates to another variable.
This is called a correlation, or a measurement of the relationship between two things.
Causation means that one variable causes another to change.
We must be careful not to assume that because two things are changing at the same time, they always exist in a cause-and-effect relationship with each other.
We know and research has shown that there is a correlation between increasing outdoor temperatures and people experiencing more sunburns. In this case, the weather caused an effect which is sunburn and it makes sense.
However, the graphic "Correlation is Not Causation" shows that there is actually a correlation between the number of children in a certain household and the number of storks nests on the roof of the house.
While it is quite an interesting correlation, it is probably safe to assume that the storks are NOT causing larger families. Perhaps it is simply that the larger homes not only have room for more storks on the roof, but they also have room for more children.
This may help explain some of the conclusions the researchers found.
For example, there was a researcher that reported that women who were diagnosed with breast cancer at a younger age said they used antiperspirants and started shaving their underarms earlier.
Since underarm odor and the development of underarm hair comes with puberty, it is probable that women who started using antiperspirants or shaving earlier in life went through puberty earlier which means they had their first menstrual period earlier. Starting periods early is already considered a risk factor for breast cancer, so was the increase in breast cancer really due to the early use of antiperspirants or the early onset of menstruation?
Conclusion
To make a somewhat simplistic analogy I can relate the importance of this topic to the process we go through when buying a new car.
Most of us are not experts on cars and the price of a car is quite an investment. We know that it is a bad idea to rely solely on the information provided by the salesperson or ads from the manufacturer. So we do as much research as possible, ask good questions and then do more research in order to make the best choice we can.
We should do the same when reading "scientific" claims about skincare products.
Looking for understanding beyond the science jargon is no easy task. But to become educated consumers we must try to see beyond the headlines, take a good look at the source of the information, gather as much information as we can and ask lots of questions in order to make informed choices in our lives.
This blog was in no way intended to be a comprehensive lesson on evaluating research. Students going into fields of science or scientific research often take an entire semester course, or more, just to learn how to read, interpret, and evaluate a piece of scientific research. I hope this information was helpful.
