Question

Propose a study in which you will perform a meta-analysis. Briefly describe the primary independent and dependent variables you will examine. What eligibility criteria should you consider in order to determine if the study should be included in the analysis

Answer 1

ANSWER-

What Is a Meta-Analysis?

There are several ways that individual studies can be summarized to help healthcare workers make decisions, including narrative reviews, systematic reviews, and meta-analyses. Both narrative and systematic reviews are both qualitative in nature.

Narrative reviews are not very rigorous, but they focus on the very basics of a topic.

Systematic reviews are more rigorous than narrative reviews; they focus on a single research question. For example, a systematic review will focus specifically on the relationship between cervical cancer and long-term use of oral contraceptives, while a narrative review may be about cervical cancer.

Meta-analyses are quantitative and more rigorous than both types of reviews. In addition to providing an overview, these papers provide a quantitative assessment of how well a treatment works or they may also provide an estimate of how much more likely a person is to develop a disease if they participate in a certain behavior.

Key Elements of Meta-Analysis-

Meta-analysis is a method used to critically evaluate evidence in an attempt to develop a single synthesis of the results and, where appropriate, to use statistical methods to combine findings from different studies into a single “pooled estimate.” The pooled estimate is also called the “overall treatment effect,” even when referring to pooling of diagnostic tests, screening studies, or other areas that are not treatments per se. The reasons to perform a meta-analysis are to:

• Increase statistical power (relative to individual studies) and determine if a treatment effect exists by combining multiple trials

• Improve the precision of the measurement of a treatment effect

• Combine data from conflicting studies and determine if a treatment effect exists

• Explore the impact on the treatment effects associated with differences in the design, conduct, analysis, and results of individual studies.

A meta-analysis begins with a systematic review of the literature, followed by a statistical analysis. Generally, it proceeds according to these steps:

1. Formulation of a study question

2. Setting inclusion and exclusion criteria for studies to be reviewed

3. Searching the literature

4. Triaging articles

5. Retrieving data from the selected studies

6. Pooling the data by applying specific statistical techniques

7. Investigating sources of differences between studies

8. Summarizing and presenting results.

Selecting the Data

To conduct a meta-analysis, one must begin by formulating the study question. A well-focused study question has 4 components that are referred to as PICO:

Population—a description of the patient population to be addressed

Intervention—a detailed description of the intervention or exposure to be investigated

Comparison—a well-defined comparison group

Outcome—a specific outcome.

For example, the following research question includes all 4 PICO elements. In adult patients with type 2 diabetes, does monotherapy with rosiglitazone improve A1c levels compared with metformin at 6 months? To minimize bias, inclusion and exclusion criteria should be identified before performing the literature search. Many of the inclusion criteria are determined as a direct result of the research question (eg, definition of the patient population or the outcome to be studied); other criteria (eg, study design) are determined by the research team.

Ideally, meta-analyses are limited to one type of study design. Significant problems are associated with meta-analyses that combine results across different study designs (eg, randomized controlled trials and case-control studies). Other inclusion and exclusion criteria may include years of publication, language of publication, and minimum study size. Because the quality of a meta-analysis is very dependent on the studies that comprise it, well-thought-out inclusion and exclusion criteria are necessary to ensure the validity of the conclusions.

Why Do a Meta-Analysis?

Medical research can be confusing. How would you make a decision if you read 30 studies that said a weight loss treatment worked and 30 that said it didn't work? What if I told you there was a better way than just flipping a coin? The reason people do meta-analyses is that research from several studies with conflicting results can be combined to make decisions about the effectiveness of a medication on a person's risks for developing a disease that is more informed than using a Magic 8-ball

How to Identify Potential Problems

Once the studies to be included in the pooled analysis have been identified, some differences will still be evident between studies. The nature and magnitude of these differences play a critical role in determining the methods to be used in the statistical analysis. Researchers have developed summary measures that quantify the degree of interstudy “heterogeneity.” Cochran's Q (based on the chi-square) test is the most common test for heterogeneity.

If “significant” heterogeneity is identified (ie, P <.20 is a typical cut-off in meta-analysis), it is important to determine whether it is valid to combine heterogeneous studies into a single pooled measure of treatment effect. For example, in a meta-analysis of hormone replacement therapy (HRT) and breast cancer, the Q test identified heterogeneity (Figure 2). One study clearly stood out—the Million Women Study, not only because it was the largest and only study conducted in Europe but also because of other important underlying clinical differences. When this study was excluded, the Q test was no longer significant, and the point estimate better reflected the pooling of similar studies.11

Figure 2

Heterogeneity, with Inclusion of the Million Women Study

WA indicates Washington State; BCDDP, Breast Cancer Detection Demonstration Project; WHS, Women's Health Study; CARES, Contraceptive And Reproductive Experiences Study; WS, Women Study.

The I2 statistic (also referred to as “inconsistency”) is an updated version of the Q test. This updated measure is designed to allow comparison of heterogeneity between meta-analyses with different numbers of pooled studies, which cannot be done with the Q test alone.12 An I2 value of less than 25% is considered good, 25% to 50% is acceptable, and more than 50% is unacceptable.

When heterogeneity is identified by the Q test or by I2, several approaches can be considered to address the differences. The first approach is to simply ignore the interstudy differences. Some investigators have argued that “one true effect” must underlie all studies on a given topic, and results should be pooled by use of a “fixed effects” model. A second method is to use a “random effects” model, which statistically accounts for heterogeneity and considers several “true effects.” (The difference between fixed and random effects models13 is beyond the scope of this review.) In general, a random effects model results in the same pooled treatment estimate but with a wider confidence interval than the fixed effects model. The majority of meta-analyses now report random effects results, and those that report only fixed effects results should be viewed with skepticism, especially those that report pooled treatment effects with marginal statistical significance.13

Another common approach to addressing heterogeneity is using meta-regression, which involves applying traditional regression techniques—and their inherent explanatory power—to meta-analysis. In meta-regression, the dependent variable is the outcome (eg, the odds ratio or the relative risk), and the independent variables are study factors, such as publication year, country, study size, or type of drug, that may reasonably be expected to contribute to interstudy differences. Stratifying and conducting subgroup meta-analyses based on these factors may help identify important sources of heterogeneity (eg, the HRT example). Although meta-regression is useful to assess heterogeneity, the results should be considered “hypothesis generating.”

The final step in meta-analysis is to look for evidence of publication bias—the exclusion of negative studies that might have influenced the conclusions. Studies published in peer-reviewed journals tend to report positive results more than negative results. In practice, researchers who find negative results often do not believe them, and editors may not as readily accept negative studies.14

Publication bias is typically evaluated by using a funnel plot. A funnel plot is a type of scatter plot in which each study's treatment effect is plotted on the x-axis, and a measure of study size is plotted on the y-axis (Figure 3). Small studies are more likely to have less precise effect estimates than large studies and are therefore more likely to be scattered along the bottom of the plot.15 In the absence of publication bias, the resulting plot should resemble an upside-down funnel, as seen in Figure 3. If publication bias is present, smaller studies that fail to show a significant effect will likely be missing, and the funnel plot will appear asymmetric (ie, the open circles in Figure 3 would be absent).15 Techniques for assessing publication bias continue to be refined,16 thus it's best to confirm that publication bias assessment was conducted in the meta-analysis rather than to assess the specific techniques used.17

Figure 3

Publication Bias Funnel Plot

The Limits of Meta-Analysis

Meta-analysis can be a powerful technique for summarizing evidence. Each meta-analysis is, in and of itself, a scientific investigation, and its quality is dependent on the methods used in carrying out the “experiment.”2 Different researchers may use different techniques, include different studies, and draw different conclusions. Like any experiment, meta-analyses are subject to bias and error, both of which may affect the validity of the conclusions and their utility for decision makers. As a result, not all meta-analyses are of equal quality. Thus “consumers” of meta-analyses—especially decision makers—must carefully assess the quality of each meta-analysis by considering the research questions asked, the methods used, the analysis and interpretation of the data, the investigation of heterogeneity, and the conclusions drawn.

Evaluating Quality

Several instruments for assessing the quality of a systematic review have been developed.15 It is important to differentiate between the quality of the reporting of a meta-analysis and the quality of the meta-analysis itself. The report may, for a number of reasons (eg, space limits, author preferences), omit important information. The QUOROM statement offers a checklist of evidence-based standards for reporting the results of meta-analyses of randomized trials.6 The QUOROM checklist—which identifies 18 key items to be included in a report and explains how to describe them—is widely used in the reporting and the evaluation of published meta-analyses. The Meta-analyses of Observational Studies in Epidemiology (MOOSE) is a similar checklist that is used to guide the reporting of meta-analyses of observational studies.18

Over the past 20 years there has been tremendous growth in the number of systematic reviews and meta-analyses. A recent study investigated the average length of time until a published systematic review requires updating.19 The authors searched the relevant literature to determine whether new evidence had been published that would either invalidate the results of a previous systematic review or would affect clinical decision-making in an important way. In almost 60% of the systematic reviews, the evidence showed that the previously published review required updating. The average length of time from publication to the emergence of new evidence was about 5.5 years, with 25% of studies becoming outdated by 2 years. This suggests that meta-analyses may have a relatively short “shelf-life,”20 and that the longer it has been since publication, the more important it is to assess whether new evidence has emerged that may change that study's conclusions.

Conclusions

Meta-analysis is a sophisticated tool for decision makers. As with all medical evidence, however, systematic reviews and meta-analyses should be regarded with due skepticism and be read critically, focusing on the elements discussed here. As meta-analyses begin to address the development of pooled estimates of cost, adverse effects, and comparative effectiveness (in addition to efficacy and effectiveness), their relevance to our understanding of how to define and measure value in healthcare will continue to grow.

​