The reproducibility crisis in research is not new to the scientific community. To demonstrate this phenomenon, cancer researchers attempted to reproduce studies published in highly rated journals and were only able to reproduce two-thirds of these studies. This leads to conflicting hypotheses and debates around particular theories. How do you move forward in your research field if the literature is conflicting?
One way to address this is to do a meta-analysis. This statistical analysis can assist you in drawing conclusions from the available data, answer questions not yet addressed, generate new hypotheses and settle debates about a particular research question.
What is a Meta-analysis?
Glass defines meta-analysis as “the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings.”
It is a way of pooling all the data pertaining to a particular question, effectively combining all the individual studies into one big study and analyzing it statistically. This increases sample size, takes more variables into account and makes a more precise analysis of the data.
Importance of Meta-analysis
It is difficult to draw conclusions when the literature is conflicting. You need to be able to make a comparison of all the literature to try and understand why the results differ. A meta-analysis can end debates about conflicting research without repeating time and resource-consuming experiments in the laboratory or in the field. For example, when you need to decide on a treatment regime for a clinical trial, a meta-analysis of all the studies can tell you which treatment was the most effective. Meta-analyses are valuable for all types of research and researchers quite frequently cite them.
Does a Meta-analysis Always Solve Conflicts?
A properly conducted meta-analysis has considered all the facts and taken all variables into consideration. They have included all the relevant studies and you can therefore trust their conclusion. However, it is not as simple as it sounds. Various questions about the studies to include in the meta-analysis need to be answered such as:
- Are the results of the study valid?
- Are the variables comparable between the studies? These are usually differences in study participants, interventions and outcomes, clinical diversity and study design.
- Is one of the data sets going to bias the results?
For example, meta-analysis has not been able to solve the ongoing debate about whether violent video games cause increased aggression and violence in young people. Studies from both sides of the debate resulted opposing conclusions. Perhaps the questions asked in these analyses were wrong. As a result, the meta-analysis increased the conflict rather than decreasing it.
Minimizing “Many Researcher Degrees of Freedom”
If two (or more) researchers perform the same meta-analysis, they must reach the same conclusion with the data set. The best approach is to decide on the research criteria up front before deciding which studies to include in the analysis.
It is important to include failed studies in a meta-analysis. If you omit a study, it may skew the results. This is tricky because journals publish positive results more frequently publish than negative ones. Some conference proceedings, graduate dissertations and clinical trial registers contain unpublished data. Overall, meta-analyses can show the bigger picture that cannot be determined in a small study.
Do you think meta-analysis is a useful research tool? Do you think the findings of a meta-analysis need to be tested empirically to verify the conclusion? Please share your thoughts with us in the comments section below.
